Object detection and pose estimation system and method using single network

Abstract

The present invention relates to an object recognition and pose estimation system using a single network and a method thereof. The present invention relates to a technology that, through a single network that simultaneously performs object recognition and pose estimation, allows for focusing on the correlation between the position and pose of an object compared to conventional algorithms divided into two stages, and is composed of a single stage for efficient operation implementation.

Description

Technology Field

[0001] The present invention relates to an object recognition and pose estimation system using a single network and a method thereof, and more specifically, to an object recognition and pose estimation system using a single network and a method thereof that can simultaneously provide object recognition and pose estimation information through relatively small computational resources by configuring a single network. Background Technology

[0002] With the advancement of computer vision utilizing deep learning, the robotics, autonomous driving, and automation system industries are growing significantly. In particular, object recognition and pose estimation algorithms are one of the core technologies in the field of computer vision, as position and pose information of objects are essential for handling objects in the robotics and augmented reality industries, and position and pose information of surrounding objects are essential for vehicles to interact with their surrounding environment and perform appropriate control based on the environment in the field of autonomous driving.

[0003] Therefore, various object estimation algorithms are being studied for object recognition and 3D pose estimation of objects.

[0004] Conventional pose estimation algorithms consist of a two-step process: detecting the location of an object in an image and estimating the pose of the object at the detected location. To utilize these pose estimation algorithms, two algorithms suitable for the industrial environment must be selected, and an environment must be configured for their operation. Configuring this environment incurs additional costs for resources, implementation, and deep learning network training. Furthermore, there is a problem where performance degradation may occur due to a lack of correlation between the object's location and its pose.

[0005] In order to resolve the problems associated with such computational load, CNN-based networks such as YOLO, which have a relatively low computational load, have been used in the past. However, there is a problem with lower accuracy compared to Transformer networks, and when using Transformer networks for object recognition and pose estimation, as described above, an additional network is required for pose estimation separately from object recognition, which still causes a problem of increased computational load.

[0006] In this regard, Korean published patent No. 10-2024-0066032 ("Method and apparatus for analyzing objects from an image using an attention network") discloses a structure for a transformer network that performs object analysis. Prior art literature

[0007] Korean Patent Publication No. 10-2024-0066032 (Publication Date: May 14, 2024) The problem to be solved

[0008] Accordingly, the present invention has been devised to solve the problems of the prior art as described above. The objective of the present invention is to provide an object recognition and pose estimation system and method using a single network that performs object recognition and pose estimation simultaneously, which allows for focusing on the correlation between the position and pose of an object compared to conventional algorithms divided into two stages, and is composed of a single stage for efficient operation implementation. means of solving the problem

[0009] The object recognition and pose estimation system using a single network according to the present invention for achieving the above-mentioned purpose is an object recognition and pose estimation system using a single network that recognizes objects and estimates poses using a Transformer-based deep learning network, comprising: a feature extraction block that extracts a feature vector of input image data; a plurality of network blocks that predict at least one of the position, type, and pose information of an object included in the image data using the feature vector and a predefined object query and pose query; and an output block that outputs the final position, final type, and final pose information of the object by sequentially passing through all network blocks constituting the plurality of network blocks. It is preferable that the pose query is input into the network block after performing a preprocessing process that applies a preprocessing projection matrix to provide continuous angular relationships.

[0010] Furthermore, it is preferable that each of the above-mentioned network blocks includes an object detection block that receives a feature vector from the object query and the feature extraction block, outputs an object detection feature vector, and predicts the location and type information of the object using the object detection feature vector, and a pose prediction block that receives the pose query and the object detection feature vector and predicts the pose information of the object.

[0011] Furthermore, it is preferable that the pose prediction block includes a query projection block that receives the object detection feature vector and outputs a query, a self attention block that receives the pose query and refines the pose information of the included object, a key projection block that receives the refined pose query and outputs a key, a value projection block that receives the refined pose query and outputs a value, and a cross attention block that receives the output query, key, and value, predicts the pose information of the object, and outputs a feature vector containing the predicted pose information.

[0012] Furthermore, it is preferable that any one network block selected among the plurality of network blocks utilizes the feature vector extracted by the feature extraction block, while the remaining network blocks, excluding the selected one network block, utilize the feature vector output by the network block placed at the front.

[0013] A method for object recognition and pose estimation using a single network that simultaneously recognizes an object and estimates a pose using a transformer-based deep learning network according to another embodiment of the present invention, wherein the method preferably comprises: a first step (S100) of extracting a feature vector of input image data in a feature extraction block; a second step (S200) of performing a preprocessing process in which a continuous angular relationship is applied to a predefined pose query in a preprocessing projection matrix; a third step (S300) of predicting at least one of the position, type, and pose information of an object included in the image data using the input feature vector, a predefined object query, and the pose query preprocessed by the second step (S200) in each of a plurality of network blocks; and a fourth step (S400) of outputting the final position, final type, and final pose information of the object using the result of sequentially passing through all network blocks constituting the plurality of network blocks in an output block.

[0014] Furthermore, it is preferable that the third step (S300) includes receiving the object query and the feature vector extracted by the first step (S100) in any one of the network blocks selected among a plurality of network blocks, outputting an object detection feature vector, predicting the location and type information of the object using the object detection feature vector, and receiving the pose query and the object detection feature vector to predict the pose information of the object.

[0015] Furthermore, the step of predicting the pose information of the object preferably includes, in each network block, a step of receiving the pose query and refining the pose information of the included object; a step of receiving the object detection feature vector and outputting the query; a step of receiving the refined pose query and outputting a key; a step of receiving the refined pose query and outputting a value; and a step of receiving the outputted query, key, and value to predict the pose information of the object and outputting a feature vector including the predicted pose information.

[0016] Furthermore, it is preferable that the third step (S300) in the network block placed after any one of the selected network blocks is performed by inputting the feature vector output from the network block placed in the preceding stage. Effects of the invention

[0017] According to the present invention, the object recognition and pose estimation system and method using a single network can be configured in a single network environment, unlike conventional pose estimation algorithms consisting of two stages, and thus has the advantage of not requiring additional resources for environment configuration.

[0018] In addition, conventional two-stage algorithms require searching for the optimal object recognition algorithm and pose recognition algorithm separately, and while a lot of cost and time are consumed for the combination of these two algorithms, the present invention has the advantage of reducing these costs by allowing object recognition and pose estimation to operate simultaneously.

[0019] In addition, there is an advantage in that additional information such as a 3D model of an object or depth information is not required for pose estimation, thereby reducing the cost and time required to obtain it. Brief explanation of the drawing

[0020] FIG. 1 is a configuration diagram of a single network proposed in an object recognition and pose estimation system using a single network and a method thereof according to an embodiment of the present invention. FIG. 2 is a configuration diagram of a network block included in a single network proposed in an object recognition and pose estimation system using a single network and a method thereof according to an embodiment of the present invention. FIG. 3 is a diagram showing the configuration of a pose prediction block included in a single network proposed in an object recognition and pose estimation system using a single network according to an embodiment of the present invention and a method thereof. FIG. 4 is an example diagram showing a preprocessing projection matrix included in a single network proposed in an object recognition and pose estimation system using a single network according to an embodiment of the present invention and a method thereof. FIG. 5 is a flowchart illustrating the learning process of a single network proposed in an object recognition and pose estimation system and method using a single network according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating the inference process of a single network proposed in an object recognition and pose estimation system using a single network and a method thereof according to an embodiment of the present invention. Specific details for implementing the invention

[0021] Hereinafter, an object recognition and pose estimation system using a single network according to the present invention, having the configuration as described above, and a method thereof will be described in detail with reference to the attached drawings. The drawings presented below are provided as examples to ensure that the concept of the present invention is sufficiently conveyed to those skilled in the art. Accordingly, the present invention is not limited to the drawings presented below and may be embodied in other forms. In addition, throughout the specification, the same reference numerals indicate the same components.

[0022] Unless otherwise defined, technical and scientific terms used herein have the meaning commonly understood by those skilled in the art to which this invention pertains, and descriptions of known functions and configurations that could unnecessarily obscure the essence of the invention are omitted in the following description and accompanying drawings.

[0023] Furthermore, a system refers to a set of components, including devices, mechanisms, and means, that are organized and interact regularly to perform necessary functions.

[0024] The field of computer vision utilizing deep learning is being applied across various industries, and in particular, 3D vision is being actively utilized in robot control, autonomous driving, virtual reality, and augmented reality industries. The most critical information in 3D vision systems is information regarding an object's position and pose; this can be utilized in robot control systems and, in the field of autonomous driving, can be used to analyze the surrounding environment of a vehicle to control it appropriately. Furthermore, in virtual and augmented reality, object pose and position information is essential for implementing physics engines that control object movement and interactions with other objects.

[0025] An object recognition and pose estimation system and method using a single network according to one embodiment of the present invention proposes a deep learning network capable of simultaneously providing object recognition and pose estimation information that is important in 3D vision. Compared to conventional two-stage networks that perform object recognition and pose estimation separately, this system requires fewer computational resources for training and is highly efficient in terms of cost for finding the optimal combination. Furthermore, by estimating object recognition and pose together, it has the advantage of focusing on the correlation between the object's position and pose compared to conventional algorithms that operate in two stages. In addition, since it can be configured as a single environment, it has the advantage of being usable on mobile and edge devices with relatively limited memory.

[0026] That is, the object recognition and pose estimation system using a single network and the method thereof according to one embodiment of the present invention preferably propose an algorithm for simultaneously estimating object recognition and pose using a transformer-based deep learning network.

[0027] For smooth explanation, FIG. 1 is a configuration diagram of a single network proposed in the object recognition and pose estimation system using a single network and method according to an embodiment of the present invention; FIG. 2 is a configuration diagram of a network block included in the single network proposed in the object recognition and pose estimation system using a single network and method according to an embodiment of the present invention; FIG. 3 is a configuration diagram of a pose prediction block included in the single network proposed in the object recognition and pose estimation system using a single network and method according to an embodiment of the present invention; FIG. 4 is an example diagram showing a preprocessing projection matrix included in the single network proposed in the object recognition and pose estimation system using a single network and method according to an embodiment of the present invention; FIG. 5 is a flowchart showing the learning process of the single network proposed in the object recognition and pose estimation system using a single network and method according to an embodiment of the present invention; and FIG. 6 is a flowchart showing the inference process of the single network proposed in the object recognition and pose estimation system using a single network and method according to an embodiment of the present invention. With reference to FIG. 1 to 6, the object recognition and pose estimation system using a single network and method according to an embodiment of the present invention will be described.

[0028] An object recognition and pose estimation system using a single network according to one embodiment of the present invention preferably has a network structure as illustrated in FIG. 1. First, each module constituting the network structure will be described, and then a learning process using the network structure and an inference (prediction) process using the network structure after learning is completed will be described.

[0029] An object recognition and pose estimation system using a single network according to one embodiment of the present invention preferably includes a feature extraction block (100), a plurality of network blocks (200), and an output block, as illustrated in FIG. 1. Each block is a transformer-based object recognition deep learning network and is configured in a computational processing means including a computer to perform operations.

[0030] The above feature extraction block (100) ) is image data from an external source (user, etc.) ) received as input, feature vector( ) is extracted. The above feature extraction block (100)( ) can use deep learning networks such as ResNet and Vision Transformer as backbone networks, and after passing through the backbone network, it passes through the Transformer Encoder to obtain the image feature vector( Extracts ).

[0031] The above network block (200) ) is the above feature vector( ), predefined object query( ) and Pose query( It is desirable to predict at least one of the location, type, and pose information of an object included in the image data using ).

[0032] The location information of the above object refers to the location of the detected object and the bounding box corresponding to the object, and the box height ( ), width( ), center point( It is composed of, and as illustrated in FIG. 1, outputs from each of the network blocks constituting the network block (200). , N is the number of pre-specified network blocks.).

[0033] Class information of the above object ( ) represents the type of detected object, and, like the location information above, is output in each of the network blocks constituting the network block (200).

[0034] Attitude information of the above object ( ) represents the attitude angle that the detected object is facing, and this is also output in each of the network blocks constituting the network block (200).

[0035] Predefined object query( ) is a learnable representation containing information about which object the network will predict—in other words, the object to be predicted—and is a predefined parameter rather than input received from the outside (user). The predefined pose query( ) is a learnable representation containing the pose information of an object predicted by the network, and is a predefined parameter rather than input received from the outside (user). The above pose query ( ) is a preprocessing projection matrix that assigns relationships with adjacent poses( It passes through ) and is input into the network block (200). That is, the pose query ( ) is the preprocessing projection matrix( After performing a preprocessing process that applies a continuous angle relationship by applying ), it is input into the network block (200), and this will be described in detail later.

[0036] In addition, as illustrated in FIG. 1, the plurality of network blocks (200) ( Any one network block selected from ) The first deployed network block is the feature vector extracted by the feature extraction block (100). ) use, but any one of the above network blocks (200)( The remaining network blocks (200) excluding ) ) is a feature vector output by the network block placed at the front ( It is desirable to use ). This is identical to a conventional object detection network, and we intend to explain it based on this.

[0037] Each of the above network blocks (200) is preferably composed of an object detection block (210) and a pose prediction block (220), as shown in FIG. 2.

[0038] The object detection block (210) is a block that predicts the location and type of an object, and uses a block utilized in a conventional object detection network. The object detection block (210) is the object query ( ) and the feature vector which is the output of the previous layer( ) takes ) as input, object detection feature vector( ) is used as the output. As illustrated in FIG. 2, the object detection block (210) is the object detection feature vector ( Using ), passing through the Bbox head and Class head composed of a Linear layer, the object's position information( ) and type information( Returns ).

[0039] In the present invention, the object detection block (210) is configured based on Deformable-DeTR, which transforms the existing Transformer's attention module into a Deformable attention module, in order to improve the learning stability, speed, and performance of the network.

[0040] The pose prediction block (220) is a block that predicts the pose of an object, and the object detection feature vector ( ) and the above pose query( ) is taken as input, and the feature vector( ) and pose query( ) is used as the output. As illustrated in FIG. 2, the pose prediction block (220) outputs the feature vector ( Using ), the object's pose information (passes through a pose head composed of a linear layer) Returns ).

[0041] At this time, the pose prediction block (220) is the feature vector ( The result output from the corresponding network block ( The Hungarian algorithm is applied to select only meaningful results from among ).

[0042] As shown in FIG. 3, the pose prediction block (220) is composed of a query projection block (221), a self attention block (222), a key projection block (223), a value projection block (224), and a cross attention block (225).

[0043] The above query projection block (221) is the object detection feature vector ( It is desirable to receive ) as input and output / generate a query.

[0044] The self-attention block (222) above is the pose query ( As a configuration for refining the information of ), the pose query ( It is desirable to receive ) as input and refine the pose information of the included / embedded object.

[0045] The above key projection block (223) is a refined pose query ( It is preferable to receive ) as input and output / generate a key, and the value projection block (224) is a refined pose query ( It is desirable to receive ) as input and output / generate a value.

[0046] The above refined pose query ( ) is a pose query input to a network block placed next to the corresponding network block, as illustrated in FIG. 1. It is used as ).

[0047] The above cross attention block (225) is the object detection feature vector ( ) and pose query( A configuration for extracting pose information through mutual information exchange between, which takes generated queries, key and value pairs as input, and extracts feature vectors ( ) is used as the output.

[0048] The feature vector (output by the above cross attention block (225)) As shown in FIG. 2, it passes through the Pose head to predict the pose of the object and is used as input to the network block placed next. By repeating this process and passing through the network block (200), more sophisticated and accurate information can be extracted.

[0049] In the present invention, a pose query ( Object recognition and pose estimation are performed simultaneously by adding a pose prediction block (220). To this end, the present invention defines the pose of an object as a classification problem instead of the conventional regression problem approach to estimate the pose of the object. In this case, although the actual angle has a continuous value, if it is defined as a classification problem, discrete values ​​are predicted regardless of continuity. To solve this, the present invention provides a pose query ( We propose a method to provide continuity to ).

[0050] That is, as described above, in the present invention, as shown in FIG. 1, a preprocessing projection matrix that assigns a relationship with adjacent poses ( Pose query that passed through ) ) is input into the above network block (200).

[0051] As shown in Fig. 4, through the preprocessing process, a pose query ( It is designed to have continuous properties. Utilizing the characteristic that a specific angle has a close relationship with adjacent angles and the relationship weakens as the angles move further apart, the projection matrix ( Constructs the above projection matrix ( )silver As a matrix, L is a user-defined pose query( It is a matrix representing the number of ) and is used to assign relationships between each angle. An angle identical to itself has a value of 0, and for angles relatively smaller than itself, negative values ​​are assigned that decrease sequentially from adjacent angles to farther angles. Conversely, for angles larger than itself, positive values ​​are assigned that increase sequentially from adjacent angles to farther angles. Additionally, the matrix is ​​constructed to maintain continuous relationships by reflecting the recursive characteristics of angles. The constructed projection matrix ( ) holds relative position information between a specific angle and another angle. The projection matrix ( ) is a matrix that has the largest value at the same angle through a Gaussian function and whose value gradually decreases according to relative position ( Constructs ). Pose query( ) is this adjacency relationship matrix ( It is transformed through ) to effectively reflect the continuity of angles, thereby increasing the precision of pose estimation.

[0052] It is preferable that the output module (300) sequentially passes through all network blocks constituting the plurality of network blocks (200) to output the final position, final type, and final pose information of the object. In order to output the final position, final type, and final pose information of the object through the output module (300) in this manner, a network learning process is required in advance.

[0053] Based on Figure 5, the learning process of the network according to the present invention will be explained.

[0054] The learning process is a user-specified Epoch Proceed once.

[0055] First, feature extraction block (100)( ), network block(200)( ), object query( ) and pose query ( Initializes ) with a random value.

[0056] specific Epoch Initialize the total loss function (L) to 0, and image data from the training database ( ), Detail information( ), object location information( ) and object class information( Receives ).

[0057] Image data ( ) is a feature extraction block (100)( Passing through ) feature vector( ) will be extracted.

[0058] Pose Query ( ) is the adjacency relationship matrix( It is converted through ).

[0059] Extracted feature vector ( ), pose query ( ) and object queries( ) is a network block (200)( It is input into ). At this time, network block (200)( ) consists of N blocks, which is the number of blocks specified by the user.

[0060] nth network block ( ) is the output of the previous layer , and Receives as input, feature vector( ), Pose Query( ), object's location information( , object type information( ) and object pose information( It will output ).

[0061] This output is the location information of the object received from the training database ( ), object type information( ) and object pose information( Compared to ), the loss function( It will produce ).

[0062] When M objects are predicted, the loss function for the object location information is as shown in Equation 1 below. It is represented as a loss function.

[0063]

[0064] In addition, when M objects are predicted, the loss function for object type information is represented as a Focal loss function as shown in Equation 2 below.

[0065]

[0066] (Here, represents the probability predicted by the model for the correct label y.)

[0067] The loss for object pose information consists of an interval classification loss function and an angle prediction loss function. To calculate the above interval classification loss function, the object pose information is divided into d discrete intervals. To calculate the difference between the discrete intervals of the predicted object pose information and the correct object pose information (object pose information received from the training database), Cross-Entropy Loss is used as shown in Equation 3 below.

[0068]

[0069] (Here, is a discrete interval of the correct object pose, and

[0070] Is This is the discrete interval of the object pose predicted in the nth network block.

[0071] In addition, the above angle prediction loss function refers to the loss function between the ground truth object pose information (object pose information received from the training database) and the predicted object pose information.

[0072] To do this, first, the angle predicted as a discrete interval ( ) must be converted into a continuous angle. For the conversion, use an equation such as Mathematical Formula 4 below.

[0073]

[0074] (Here, d is the number of discrete intervals, and

[0075] is the predicted interval Defined in,

[0076] Is Defined in.)

[0077] The above angle prediction loss function uses SmoothL1 Loss between the transformed angle and the correct angle, which is calculated as shown in Equation 5 below.

[0078]

[0079] (Here, is the predicted object pose angle, and

[0080] It is the actual object pose angle, and

[0081] is fixed to a value of 1.)

[0082] Through this process, the loss function in the nth network block ( ) can be defined as shown in Equation 6 below, and the total loss function (L_total) is equal to the sum of the loss functions of all network blocks as shown in Equation 7 below.

[0083]

[0084]

[0085] After passing through all network blocks in this way, the feature extraction block (100)( ), network block(200)( ), object query( ) and pose query( The parameters of ) are updated through the Backpropagation algorithm.

[0086] This is a user-specified Epoch After repeating the process a certain number of times, the learning process ends.

[0087] FIG. 6 is a flowchart illustrating a method for object recognition and pose estimation using a single network that simultaneously estimates object recognition and pose using a transformer-based deep learning network according to an embodiment of the present invention, and corresponds to the inference process of the single network proposed in the object recognition and pose estimation system using a single network according to an embodiment of the present invention and the method thereof.

[0088] First, the feature extraction block (100) learned through the learning process described above ( ), network block(200)( ), object query( ) and pose query( ) will be received.

[0089] Image data from the user ( ) receiving, and the received image data ( ) is input into the above feature extraction block (100).

[0090] Afterwards, in the first step (S100), in the feature extraction block (100), the feature vector of the input image data ( ) will be extracted.

[0091] In the second step (S200), the received pose query ( ) performs a preprocessing process that assigns continuous angle relationships. Specifically, the pose query ( ) is a preprocessing projection matrix that assigns relationships with adjacent poses( It is converted by passing through ).

[0092] That is, as shown in Fig. 4, through a preprocessing process, a pose query ( It is designed to have continuous properties. Utilizing the characteristic that a specific angle has a close relationship with adjacent angles and the relationship weakens as the angles move further apart, the projection matrix ( Constructs the above projection matrix ( )silver As a matrix, L is a user-defined pose query( It is a matrix representing the number of ) and is used to assign relationships between each angle. An angle identical to itself has a value of 0, and for angles relatively smaller than itself, negative values ​​are assigned that decrease sequentially from adjacent angles to farther angles. Conversely, for angles larger than itself, positive values ​​are assigned that increase sequentially from adjacent angles to farther angles. Additionally, the matrix is ​​constructed to maintain continuous relationships by reflecting the recursive characteristics of angles. The constructed projection matrix ( ) holds relative position information between a specific angle and another angle. The projection matrix ( ) is a matrix that has the largest value at the same angle through a Gaussian function and whose value gradually decreases according to relative position ( Constructs ). Pose query( ) is this adjacency relationship matrix ( It is transformed through ) to effectively reflect the continuity of angles, thereby increasing the precision of pose estimation.

[0093] In the third step (S300), the plurality of network blocks (200) ( In each of ) the input feature vector and object query( ) and the pose query preprocessed by the second step (S200) above ( It is desirable to predict at least one of the location, type, and pose information of an object included in the image data using ).

[0094] The above third step (S300) is performed through the object detection block (210), the object query ( ) and the feature vector which is the output of the previous layer( ) takes ) as input, object detection feature vector( ) is used as the output. As illustrated in FIG. 2, the object detection block (210) is the object detection feature vector ( Using ), passing through the Bbox head and Class head composed of a Linear layer, the object's position information( ) and type information( Returns ).

[0095] The above third step (S300) also includes, through the pose prediction block (220), the object detection feature vector ( ) and the above pose query( ) is taken as input, and the feature vector( ) and pose query( ) is used as the output. As illustrated in FIG. 2, the pose prediction block (220) outputs the feature vector ( Using ), the object's pose information (passes through a pose head composed of a linear layer) Returns ).

[0096] At this time, the pose prediction block (220) is the feature vector ( The result output from the corresponding network block ( The Hungarian algorithm is applied to select only meaningful results from among ).

[0097] At this time, the pose prediction block (220) through the self-attention block (222), the pose query ( It receives ) as input and refines the pose information of the included / embedded object.

[0098] Afterwards, through the query projection block (221), the object detection feature vector ( ) is received as input, a query is output / generated, and through the key projection block (223), a refined pose query ( ) receives input, outputs / generates a key, and through the value projection block (224), a refined pose query ( It receives ) as input and outputs / generates a value.

[0099] The above refined pose query ( ) is a pose query input to a network block placed next to the corresponding network block, as illustrated in FIG. 1. It is used as ).

[0100] Additionally, the pose prediction block (220) receives the generated query, key, and value as input through the cross attention block (225), and extracts information about the pose of the detected object based on the correlation between the object detection information and the inherent pose information, and a feature vector ( ) is used as the output. The feature vector (which is the output of the cross attention block (225)) As shown in FIG. 2, it passes through the Pose head to predict the pose of the object and is used as input to the network block placed next. By repeating this process and passing through the network block (200), more sophisticated and accurate information can be extracted.

[0101] Accordingly, the above third step (S300) is performed by inputting the feature vector output from the network block placed in the front end.

[0102] In the fourth step (S400), the output module (300) sequentially passes through all network blocks constituting the plurality of network blocks (200) to output the final location, final type, and final attitude information of the object. That is, the output that has passed through all network blocks (200) Use only ) as the final prediction / inference result.

[0103] To verify the performance of the present invention, the Carvana database was used. The Carvana database contains various types of vehicle objects and various vehicle attitudes. 4,070 images were randomly extracted for training, and 1,018 images were used as verification images to evaluate the performance of the present invention. In addition, the optimal model was searched by changing the Cost view, a coefficient related to the object attitude loss function, and the loss function itself. The azimuth angle, which is a key angle among the object attitude information, was estimated; the discrete interval of the angle was set to 16, and the model training was conducted for 50 epochs.

[0104] Performance measurements were performed on mAP, which estimates the position information of an object, and mAVP, which estimates the pose information of an object, and the results are shown in Tables 1 and 2 below. mAVP is an Average Precision metric for cases where the IoU between the object's predicted position box and the ground truth box is greater than or equal to a specific threshold, and the predicted object pose matches the ground truth pose.

[0105]

[0106]

[0107] The best performance was achieved when the Cost view was 0.5, and the object recognition performance was 0.97 mAP based on an IoU of 0.5, while the pose estimation performance was 0.705 mAVP based on an IoU of 0.5. Additionally, among the pose estimation loss functions, the highest performance was observed when SmoothL1 Loss was used instead of L1 Loss for the angle regression loss function. As shown in these performance verification results of the present invention, an optimal network was discovered through the present invention, and good performance was confirmed in both mAP and mAVP metrics, so it is expected that it can be applied to actual industrial sites.

[0108] Meanwhile, an object recognition and pose estimation system using a single network and a method thereof according to one embodiment of the present invention may be implemented in the form of program instructions that can be executed through various means of electronically processing information and recorded on a storage medium. The storage medium may include program instructions, data files, data structures, etc., either individually or in combination.

[0109] Program instructions recorded on a storage medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of software. Examples of storage media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a device that processes information electronically using an interpreter, such as a computer.

[0110] As described above, the present invention has been explained with specific details such as specific constituent elements and limited exemplary drawings; however, this is provided merely to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above-mentioned exemplary embodiment. Those skilled in the art can make various modifications and variations from this description.

[0111] Accordingly, the scope of the present invention should not be limited to the described embodiments, and all things equivalent to or having equivalent variations to the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention. Explanation of the symbols

[0112] 100 : Feature extraction block 200: Network block 210: Object Detection Block 220 : Pose Prediction Block 221 : Query projection block 222 : Self-Attention Block 223 : Key projection block 224 : Value Projection Block 225 : Cross attention block 300 : Output block

Claims

Claim 1 An object recognition and pose estimation system using a single network that recognizes objects and estimates poses using a transformer-based deep learning network, comprising: a feature extraction block that extracts feature vectors of input image data; a plurality of network blocks that predict at least one of position, type, and pose information of an object included in the image data using the feature vectors and predefined object queries and pose queries; and an output block that sequentially passes through all network blocks constituting the plurality of network blocks to output the final position, final type, and final pose information of the object, wherein the pose query is input to the network blocks after performing a preprocessing process that applies a preprocessing projection matrix to provide continuous angular relationships. Claim 2 An object recognition and pose estimation system using a single network, wherein each of the above network blocks receives the object query and the feature vector as input, outputs an object detection feature vector, and predicts the location and type information of the object using the object detection feature vector; and a pose prediction block receives the pose query and the object detection feature vector as input and predicts the pose information of the object. Claim 3 In claim 2, the pose prediction block comprises: a query projection block that receives the object detection feature vector and outputs a query; a self attention block that receives the pose query and refines the pose information of the included object; a key projection block that receives the refined pose query and outputs a key; a value projection block that receives the refined pose query and outputs a value; and a cross attention block that receives the output query, key, and value, predicts the pose information of the object, and outputs a feature vector including the predicted pose information. This describes an object recognition and pose estimation system using a single network. Claim 4 An object recognition and pose estimation system using a single network, wherein, in claim 3, any one network block selected among the plurality of network blocks uses the feature vector extracted by the feature extraction block, and the remaining network blocks excluding the any one network block use the feature vector output by the network block placed in the front. Claim 5 A method for object recognition and pose estimation using a single network that simultaneously recognizes an object and estimates a pose using a transformer-based deep learning network, comprising: a first step (S100) of extracting a feature vector of input image data in a feature extraction block; a second step (S200) of performing a preprocessing process in which a predefined pose query is given a continuous angular relationship in a preprocessing projection matrix; a third step (S300) of predicting at least one of the position, type, and pose information of an object included in the image data using an input feature vector, a predefined object query, and a pose query preprocessed by the second step (S200) in each of a plurality of network blocks; and a fourth step (S400) of outputting the final position, final type, and final pose information of the object using the result of sequentially passing through all network blocks constituting the plurality of network blocks in an output block. Claim 6 In claim 5, the third step (S300) comprises: receiving the object query and the feature vector extracted by the first step (S100) in any one of a plurality of network blocks selected, outputting an object detection feature vector, and predicting the location and type information of the object using the object detection feature vector; and receiving the pose query and the object detection feature vector to predict the pose information of the object; a method for object recognition and pose estimation using a single network. Claim 7 In claim 6, the step of predicting pose information of the object comprises, in each network block, a step of receiving the pose query and refining the pose information of the included object; a step of receiving the object detection feature vector and outputting the query; a step of receiving the refined pose query and outputting a key; a step of receiving the refined pose query and outputting a value; and a step of receiving the outputted query, key, and value to predict the pose information of the object and outputting a feature vector including the predicted pose information; an object recognition and pose estimation method using a single network. Claim 8 In claim 7, the third step (S300) in a network block placed after any one network block selected among a plurality of network blocks is performed by inputting the feature vector output from the network block placed in the preceding step, in an object recognition and pose estimation method using a single network.

Images