Method and apparatus for processing MIMO fmcw radar signal

Abstract

An object recognition method and apparatus are disclosed. An object recognition method according to one embodiment of the present disclosure may include: receiving first point cloud data of an i-1th frame among consecutive frames; outputting information about at least one object by extracting features from the first point cloud data; performing tracking of at least one object in the i-th frame based on the information about at least one object to predict a point cloud according to the position of at least one object in the i-th frame; merging the point cloud data according to the position of at least one object in the i-th frame with the first point cloud data to generate enhanced second point cloud data; and performing object recognition based on the second point cloud data.

Description

Technology Field

[0001] The present disclosure relates to an object recognition method and apparatus capable of performing object recognition by augmenting the point cloud of a 4D imaging radar based on relative velocity. Background Technology

[0002] With the advancement of artificial intelligence technology, image processing-based object recognition and tracking technology is being utilized in various fields.

[0003] In particular, there is increasing demand for models capable of recognizing and tracking objects in applications such as autonomous vehicles and security monitoring devices. In autonomous vehicles, the shape of the surroundings can be represented as a point cloud using information collected by radar or lidar sensors.

[0004] Meanwhile, recently, 4D imaging radar has stronger characteristics in bad weather compared to lidar, so there is a trend to equip autonomous vehicles with 4D imaging radar instead of lidar.

[0005] However, if the number of points measured at once is 200 when using LiDAR to measure an object, radar outputs very few points, such that only 10 to 13 points are measured on the same object.

[0006] Accordingly, since the number of acquired points is small when using 4D imaging radar to recognize specific objects, the object recognition rate may be low. Therefore, there is a need for a model that can improve the accuracy of object tracking and recognition to the point where it can be applied to real-time object recognition while using 4D imaging radar.

[0007] The aforementioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the process of deriving the present invention, and it cannot be considered as prior art disclosed to the general public prior to the filing of the present invention. Prior art literature

[0008] Prior Art 1: Korean Published Patent Application No. 10-2021-0107119 (Aug. 31, 2021) The problem to be solved

[0009] One objective of the embodiment of the present disclosure is to predict the position of a corresponding point in the next frame based on the position value and relative velocity of each point of an object in the current frame, and to perform object recognition by augmenting the point cloud of a 4D imaging radar using both the predicted position value of the point in the next frame and the position value of the point recognized in the current frame.

[0010] One objective of the embodiment of the present disclosure is to compensate for the problem of insufficient number of measured point cloud data compared to LiDAR by performing point cloud augmentation of an object in the current frame through the prediction of the object's point cloud in the next frame.

[0011] One objective of the embodiments of the present disclosure is to enable the replacement of the functions of LiDAR by improving low-complexity resolution as well as being robust against adverse weather conditions.

[0012] The purpose of the embodiments of the present disclosure is not limited to the problems mentioned above, and other unmentioned purposes and advantages of the present invention may be understood from the following description and will be more clearly understood by the embodiments of the present invention. Furthermore, it will be understood that the purposes and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims. means of solving the problem

[0013] An object recognition method according to one embodiment of the present disclosure may include: receiving first point cloud data of an i-1th frame among consecutive frames; outputting information about at least one object by extracting features from the first point cloud data; performing tracking of at least one object in the i-th frame based on the information about at least one object to predict a point cloud according to the position of at least one object in the i-th frame; merging the point cloud data according to the position of at least one object in the i-th frame with the first point cloud data to generate enhanced second point cloud data; and performing object recognition based on the second point cloud data.

[0014] In addition to this, other methods for implementing the present invention, other systems, and computer-readable recording media storing a computer program for executing said methods may be further provided.

[0015] Other aspects, features, and advantages other than those described above will become clear from the following drawings, claims, and detailed description of the invention. Effects of the invention

[0016] According to an embodiment of the present disclosure, the position of a corresponding point in the next frame is predicted based on the position value and relative velocity of each point of an object in the current frame, and the object recognition rate can be improved by augmenting the point cloud of the 4D imaging radar using both the predicted point position value in the next frame and the point position value recognized in the current frame.

[0017] In addition, by performing point cloud augmentation of an object in the current frame through the prediction of the object's point cloud in the next frame, the problem of insufficient point cloud data being measured compared to LiDAR can be compensated for.

[0018] In addition, according to an embodiment of the present disclosure, by improving low-complexity resolution as well as being robust against adverse weather conditions, the resolution weakness caused by the small number of points of conventional radar can be improved, thereby replacing the function of lidar.

[0019] The effects of the present disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0020] FIG. 1 is a drawing for explaining an object recognition system environment according to one embodiment. FIG. 2 is an illustrative diagram for explaining an environment in which an object recognition method and device according to one embodiment are used. FIG. 3 is a block diagram schematically showing the configuration of an object recognition device according to one embodiment. FIG. 4 is a drawing for explaining point cloud enhancement according to one embodiment. FIG. 5 is a flowchart illustrating an object recognition method according to one embodiment. Specific details for implementing the invention

[0021] The advantages and features of the present disclosure and the methods for achieving them will become clear by referring to the embodiments described in detail together with the accompanying drawings.

[0022] However, the present disclosure is not limited to the embodiments presented below, but can be implemented in various different forms and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure. The embodiments presented below are provided to make the present disclosure complete and to fully inform those skilled in the art of the scope of the disclosure. In describing the present disclosure, detailed descriptions of related prior art are omitted if it is determined that such detailed descriptions may obscure the essence of the present disclosure.

[0023] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. Terms such as “first,” “second,” etc., may be used to describe various components, but the components should not be limited by such terms. Such terms are used solely for the purpose of distinguishing one component from another.

[0024] Hereinafter, embodiments according to the present disclosure will be described in detail with reference to the attached drawings. In describing with reference to the attached drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0025] FIG. 1 is a diagram illustrating an object recognition system environment according to one embodiment, and FIG. 2 is an example diagram illustrating an environment in which an object recognition method and device according to one embodiment are used.

[0026] Referring to FIGS. 1 and 2, an object recognition system (1) according to one embodiment relates to a technique for predicting the position of a point in the current frame using the position value and relative velocity of each point in the previous frame of an object in order to increase the recognition rate of an object in an object recognition model through radar, and can increase the number of points that can be used in the object recognition model by using both the predicted point position value and the point position value recognized in the current frame.

[0027] Meanwhile, although FIGS. 1 and 2 describe the use of an object recognition method and device to detect the surrounding environment in an autonomous vehicle, the object recognition method and device of the present disclosure are not limited to use in the field of autonomous vehicles, and can be utilized in all other fields where the function of recognizing and tracking objects is required.

[0028] Referring to FIG. 1, an object recognition system (1) of one embodiment may include an object recognition device (200), a server (300), and a network (400).

[0029] Additionally, referring to FIG. 2, the autonomous vehicle (10) may include imaging radars (110, 120, 130, 140) for detecting the surrounding environment. Although FIG. 2 is shown as detecting only the front and rear of the vehicle (10), the imaging radars (110, 120, 130, 140) may be configured to detect the entire surrounding environment, including the sides of the vehicle (10), through changes in operation or additional placement.

[0030] Imaging radar (110, 120, 130, 140) is a radar capable of obtaining information not only about the presence of a target but also about its shape, and may include a space object identification radar (SOI) or a synthetic aperture radar (SAR).

[0031] Information about the surrounding environment obtained from imaging radars (110, 120, 130, 140) can be represented as points and processed by a processor as point cloud data.

[0032] On the other hand, compared to the output of LiDAR, imaging radar has lower resolution, resulting in a smaller number of points in its point cloud data. Due to these characteristics, unlike LiDAR data, which is typically downsampled to reduce computational load and memory usage, imaging radar performs upsampling to increase the number of points, thereby improving the accuracy of object recognition and tracking.

[0033] Traditional radar systems are configured to scan roads across a horizontal plane and can identify three-dimensional information about objects (distance to the object, direction of movement of the object, relative speed of the object).

[0034] Newly used 4D imaging radar systems can add a new dimension of information called vertical information. These devices are also labeled as imaging radars due to the richness of the data they acquire. In other words, imaging radars can detect a very large number of different reflection points that allow for the construction of an image using both horizontal and vertical data.

[0035] In one embodiment, the object recognition system (1) can acquire point cloud data through a 4D imaging radar, and such point cloud data may include information about the location, movement speed, movement direction, and height of an object detected by the 4D imaging radar.

[0036] If point cloud data acquired through a 4D imaging radar is properly processed and utilized according to the method described below, recognition and tracking of objects in the surrounding environment of a device equipped with a 4D imaging radar can be achieved with high accuracy.

[0037] In other words, the point cloud of a 4D imaging radar is significantly smaller than the point cloud of a LiDAR, making it difficult to classify objects. Therefore, to solve the problem of low recognition rates, it is necessary to increase the number of points for specific objects.

[0038] Accordingly, in one embodiment, the position of an object in the next frame can be predicted using position values ​​and relative velocity values ​​from the point cloud data of a specific object. By applying a point cloud augmentation technique of a specific object in the current frame through prediction, the problem of insufficient number of measured point cloud data compared to LiDAR, which is a disadvantage of 4D imaging radar, can be compensated for.

[0039] Therefore, in one embodiment, the disadvantage of having fewer point clouds, although stronger than lidar in adverse weather conditions, can be compensated for, thereby improving the usability of radar for future autonomous vehicles.

[0040] In one embodiment, the object recognition device may be implemented as an internal component (200) such as a car, airplane, or video surveillance device, or in another embodiment, part or all thereof may be implemented as a server (300). That is, the object recognition device may be a device attached to a device requiring object recognition or implemented as a module, or it may be a server positioned to enable remote communication away from the device requiring object recognition.

[0041] When implemented as a server (300), point cloud data or object recognition results obtained from the imaging radar (100) in a vehicle, airplane, video surveillance device, etc. are transmitted to the server (300), and the server (300) can enhance the point cloud data, recognize an object based on the enhanced point cloud data, or modify the object recognition results. The following embodiments are described on the premise that the object detection device is implemented as an internal component (200) of a vehicle, airplane, video surveillance device, etc.

[0042] That is, in one embodiment, the object recognition device (200) may be implemented by the object recognition device (200) and / or the server (300), wherein the server (300) may be a server for the object recognition device (200) to operate the object recognition system (1) or a server that implements a part or the whole of the object recognition device (200).

[0043] In one embodiment, the server (300) may be a server that controls the operation of an object recognition device (200) for an overall process of tracking point cloud data acquired for at least one object from an imaging radar (100), predicting the direction of movement of the object in the next frame using the position value and relative velocity value of the data, and augmenting the point cloud of a specific object by adding the predicted point cloud position to the point in the current frame.

[0044] Additionally, the server (300) may be a database server that provides data for operating the object recognition device (200). Furthermore, the server (300) may include a web server, an application server, or a deep learning network providing server.

[0045] And the server (300) may include a big data server and an AI server required to apply various artificial intelligence algorithms, and a computation server that performs computations of various algorithms.

[0046] In addition, in this embodiment, the server (300) may include the servers described above or network with such servers. That is, in this embodiment, the server (300) may include the web server and AI server mentioned above or network with such servers.

[0047] In an object recognition system (1), an object recognition device (200) and a server (300) may be connected by a network (400). Such a network (400) may include wired networks such as LANs (local area networks), WANs (wide area networks), MANs (metropolitan area networks), and ISDNs (integrated service digital networks), or wireless networks such as wireless LANs, CDMA, Bluetooth, and satellite communication, but the scope of the present invention is not limited thereto. Additionally, the network (400) may transmit and receive information using short-range communication and / or long-range communication.

[0048] Additionally, the network (400) may include connections of network elements such as hubs, bridges, routers, switches, and gateways. The network (400) may include one or more connected networks, such as a multi-network environment, including a public network such as the Internet and a private network such as a secure corporate private network. Access to the network (400) may be provided through one or more wired or wireless access networks. Furthermore, the network (400) may support an Internet of Things (IoT) network and / or 5G communication that exchanges and processes information between distributed components, such as objects.

[0049] FIG. 3 is a block diagram schematically showing the configuration of an object recognition device according to one embodiment.

[0050] As illustrated in FIG. 2, a 4D imaging radar (100) can be installed on a device that requires awareness of the surrounding environment and collect point cloud data of the surrounding environment.

[0051] The point cloud data collected in this way can be transmitted to an object recognition device (200). The object recognition device (200) may include a processor (210) and a memory (220), and a display (230) may be included in the object recognition device (200) or provided as a separate component.

[0052] The processor (210) can control the overall operation of the object recognition device (200). Specifically, the processor (210) is connected to the configuration of the object recognition device (200) including memory (220) and can control the overall operation of the object recognition device (200) by executing at least one command stored in the memory (220).

[0053] The processor (210) can be implemented in various ways. For example, the processor (210) can be implemented as at least one of an Application Specific Integrated Circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), or a digital signal processor (DSP).

[0054] The processor (210) is a type of central processing unit that can control the operation of the entire object recognition device (200) by running control software loaded in memory (220). The processor (210) may include all types of devices capable of processing data. Here, 'processor' may refer to a data processing device embedded in hardware, having a physically structured circuit to perform a function expressed by code or instructions included in a program, for example.

[0055] The memory (220) can store various information required for the operation of the object recognition device (200) and / or the control (operation) of the server (300), and can store control software, and may include a volatile or non-volatile recording medium.

[0056] The memory (220) is connected to one or more processors (210) and can store codes that cause the processor (210) to control the object recognition device (200) and / or server (300) when executed by the processor (210).

[0057] Here, the memory (220) may be a non-transient storage medium such as a magnetic storage medium or a flash storage medium, or a transient storage medium such as RAM, but the scope of the present invention is not limited thereto. Such memory (220) may include internal memory and / or external memory, and may include volatile memory such as DRAM, SRAM, or SDRAM, non-volatile memory such as OTPROM (one time programmable ROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, NAND flash memory, or NOR flash memory, flash drives such as SSD, CF (compact flash) card, SD card, Micro-SD card, Mini-SD card, Xd card, or memory stick, or storage devices such as HDD. In addition, information related to an algorithm for performing learning according to the present disclosure may be stored in the memory (220). In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (220), and the information stored in the memory (220) may be updated as it is received from a server or external device or input by a user.

[0058] The object recognition device (200) may include a communication unit (not shown), and the communication unit may provide a communication interface necessary to provide transmission and reception signals between external devices in the form of packet data in conjunction with a network. Additionally, the communication unit may be a device including hardware and software necessary to transmit and receive signals, such as control signals or data signals, through a wired or wireless connection with another network device.

[0059] That is, the processor (210) can receive various data or information from an external device connected through a communication unit, and can also transmit various data or information to the external device.

[0060] Meanwhile, the object recognition device (200) may include a user interface (not shown), and in one embodiment, the user interface may include an input interface into which user requests and commands for controlling the operation of the object recognition device (200) (e.g., changing network parameters, changing network learning conditions, etc.) are input.

[0061] In addition, in one embodiment, the user interface may include an output interface that outputs object recognition results. That is, the user interface may output results according to user requests and commands. The input interface and the output interface of such a user interface may be implemented in the same interface.

[0062] Meanwhile, the result calculated by the object recognition device (200) can be output through the display (230). For example, an object recognized and estimated by the object recognition device (200) can be displayed and shown to the user through the display (230).

[0063] In one embodiment, the result calculated by the object recognition device (200) may be used to control the operation of a device connected to the object recognition device (200) without being displayed to the user.

[0064] FIG. 4 is a drawing for explaining point cloud enhancement according to one embodiment.

[0065] The processor (210) and memory (220) of the object recognition device (200) are electrically connected, and the processor (210) can receive the first point cloud data (which may be expressed as initial point cloud data) of the i-1th frame among consecutive frames according to instructions stored in the memory (220).

[0066] The term "consecutive frames" means that point cloud data regarding the surrounding environment at one point in time is received according to the data collection rate of the imaging radar (100) (point cloud data of the i-1th frame), and then point cloud data regarding the surrounding environment at the next point in time (point cloud data of the i-th frame) is received, so that the processor (210) can identify an object in the previous frame and track where the identified object is located in the next frame. That is, in one embodiment, the point cloud data of the i-th frame may refer to the cloud data of the i-th frame among the total point cloud data received according to the data collection rate.

[0067] The processor (210) can predict the point cloud data of the next frame based on the received first point cloud data and merge the predicted point cloud data with the first point cloud data to generate enhanced second point cloud data. As described above, since the point cloud data acquired by the imaging radar may lack points to accurately perform object recognition, the point cloud data can be augmented according to the method described below, as shown in FIG. 4.

[0068] The processor (210) can output information about at least one object by extracting features from the first point cloud data using a transformer model.

[0069] Features may also be used to determine the identity of objects to be tracked through recognition and identification of objects. Here, features may refer to features of input point cloud data required for deep learning training.

[0070] The processor (210) can predict a point cloud for at least one object in the next frame by performing tracking of at least one object based on information about at least one object.

[0071] That is, since information such as the relative velocity, position value, and cross section of a measured point is recorded in the 4D imaging radar (100), the processor (210) can predict the direction of movement of at least one object in the i-th frame based on the relative velocity value and position value of at least one object in the i-1th frame. In addition, the processor (210) can predict the position value of the point cloud data in the i-th frame for at least one object according to the predicted direction of movement. Here, i is a natural number greater than or equal to 2.

[0072] The processor (210) can generate enhanced second point cloud data by merging point cloud data according to the position in the i-th frame for at least one object with the first point cloud data.

[0073] And the processor (210) can perform object recognition based on the second point cloud data.

[0074] Meanwhile, in another embodiment, for example, the processor (210) can perform tracking of at least one object in the next frame by applying information about at least one object to a Kalman filter.

[0075] The Kalman filter is a minimum variance state estimation algorithm for state estimation in linear dynamic systems. The Kalman filter for tracking moving objects in an image can predict the most likely object location in the current frame based on the tracking result of the previous frame, search for a target in the adjacent area of ​​the predicted location, and if a target exists in the search area, continuously track to predict the location of the target in the next frame. By using the Kalman filter, the location and velocity of a moving object can be accurately predicted, and the parameters of the moving object can be estimated. Through the above method, the processor (210) can track where an object from the previous frame is located in the next frame.

[0076] FIG. 5 is a flowchart illustrating an object recognition method according to one embodiment.

[0077] Referring to FIG. 5, the object recognition process of the processor (210) will be described in more detail. The object recognition method of FIG. 5 can be performed by the processor (210) of the object recognition device (200).

[0078] In step S100, the processor (210) receives the first point cloud data of the i-th frame among consecutive frames.

[0079] Meanwhile, an object recognition method according to one embodiment may use a deep learning model capable of object recognition by utilizing point cloud data of a 4D imaging radar based on the structure of a transformer model.

[0080] Here, the Transformer model is a neural network that learns context and meaning by tracking relationships within sequential data. It applies a continuously evolving mathematical technique known as attention or self-attention to detect even subtle changes in the meaning of separated data elements depending on their relationships.

[0081] Because the Transformer mathematically identifies patterns between elements, it eliminates the need to train neural networks on large-scale, labeled datasets. This enables the use of trillions of images and petabytes of text data in the web and enterprise databases. Furthermore, since the computations used by the Transformer are suitable for parallel processing, the execution speed of the model can also be accelerated.

[0082] As with most neural networks, the Transformer model essentially corresponds to a large encoder / decoder block that processes data. The Transformer tags data elements transmitted to and from the network using position encoders, and following these tags, the attention unit can compute a type of algebraic map that shows the relationships between the data elements. Attention queries can typically be executed in parallel by computing a matrix of equations known as multi-headed attention.

[0083] That is, in one embodiment, by using a transformer model, the processor (210) can recognize objects in the same or similar way as objects recognized by humans with respect to point cloud data acquired from a 4D imaging radar.

[0084] However, depending on the embodiment, CNN and RNN may be applied to recognize objects in point cloud data.

[0085] The transformer model is configured to embed point cloud data. In the case of a 4D imaging radar object tracking transformer model according to one embodiment, since point cloud data of the 4D imaging radar is used, it may include a structure for embedding points in a 5-dimensional vector space.

[0086] Here, since the point itself includes location information as coordinates, there may not be a need to embed the location information separately.

[0087] Accordingly, the processor (210) can embed the first point cloud data into a five-dimensional vector space. Here, embedding the point cloud data into a five-dimensional vector space may mean using five values ​​of x, y, z, Doppler, and power from the raw data of the 4D imaging radar as inputs.

[0088] Meanwhile, in order to further increase the accuracy of object recognition and tracking of the 4D imaging radar, a process of enhancing point cloud data is required. Accordingly, in one embodiment, point cloud data can be enhanced through steps S200 to S400.

[0089] In step S200, the processor (210) outputs information about at least one object by extracting features from the first point cloud data.

[0090] For example, the processor (210) can extract features by applying a K-nearest neighbor search algorithm and using the self-attention of the transformer model to perform operations using a reference point and k points adjacent to it.

[0091] Subsequently, the processor (210) can output a box classification result and a bounding box regression through a region proposal network using the extracted features as input. Here, box classification refers to classifying whether an object exists, and bounding box regression refers to regressing the coordinates of a bounding box where an object is likely to be located. The output obtained by applying a region proposal network (RPN) to the features extracted through the self-attention module described above may be box classification and bounding box regression.

[0092] The processor (210) can take the extracted feature as input and output the box classification result and bounding regression through the RPN. The input value of the RPN is a feature map extracted from the previous model. An nxn window can be slid over the feature map to generate a region proposal.

[0093] And in step S300, the processor (210) performs tracking of at least one object in the i-th frame based on information about at least one object and predicts a point cloud according to the position of at least one object in the i-th frame.

[0094] At this time, the processor (210) can predict the direction of movement of at least one object in the i-th frame based on the relative velocity value and position value of at least one object in the i-1 frame. And the processor (210) can predict the position value of the point cloud data of at least one object in the i-th frame according to the predicted direction of movement.

[0095] Meanwhile, as described above, when the object location in the next frame is predicted based on the Kalman filter, the processor (2210) can predict the object location in the next frame and implement object tracking by inputting the output of the RPN into the Kalman filter and performing calculations while repeating the prediction and update process. That is, the processor (210) can predict the object location in the next frame by inputting the box classification result and bounding regression into the Kalman filter and performing calculations while repeating the prediction and update process.

[0096] Next, in step S400, the processor (210) generates enhanced second point cloud data by merging the point cloud data according to the position in the i-th frame for at least one object with the first point cloud data.

[0097] Meanwhile, the processor (210) can change the data collection frame rate based on the relative speed of at least one object in the i-1th frame.

[0098] At this time, the processor (210) can change the data collection frame rate to be longer than the set value if the relative speed for at least one object in the i-1th frame is less than the reference value, and change the data collection frame rate to be shorter than the set value if the relative speed is greater than the reference value.

[0099] This is to improve the accuracy of prediction and actual detection results for point cloud data in the next frame by shortening the data collection frame rate, as the faster the relative speed of the object, the greater the change of the object in the next frame.

[0100] Additionally, in one embodiment, the processor (210) can calculate accuracy based on the result of comparing the predicted position value and the measured position value for the point cloud data in the i-th frame for at least one object.

[0101] And the processor (210) can adjust the ratio of merging the location value of the point cloud data in the i-th frame for at least one predicted object into the first point cloud data based on accuracy.

[0102] For example, the processor (210) compares the predicted location value of the cloud point data in the next frame with the measured location value of the actual cloud point data in the next frame, and if the accuracy of the comparison result is below a threshold, it may merge the first point cloud data by reflecting only a portion of the predicted location value. This is to reduce the error by reducing the reflection ratio of the predicted data as it is determined that the predicted location value is inaccurate.

[0103] Additionally, in one embodiment, the processor (210) can determine whether there is a region of interest with respect to the point cloud data in the i-th frame for at least one object.

[0104] And the processor (210) can adjust the ratio of merging the location value of the point cloud data in the i-th frame for the predicted at least one object into the first point cloud data based on whether there is a region of interest of at least one object.

[0105] For example, the processor (210) may determine whether at least one object is a region of interest, and for regions that are not regions of interest, it may not merge the predicted location values ​​into the first point cloud data or reduce the ratio thereof. Alternatively, for example, when adjusting the reflection ratio of the predicted data based on accuracy, for regions that are not regions of interest where the accuracy is determined to be below a threshold, the predicted location values ​​may not be merged into the first point cloud data at all.

[0106] This is intended to improve the accuracy of object recognition by enabling point cloud data enhancement to be performed more intensively on the region of interest.

[0107] Meanwhile, in one embodiment, the processor (210) can generate new points based on the i-th frame for all points of the first point cloud and can determine whether the density of the point cloud data containing the new points based on the i-th frame has reached a set density.

[0108] And if the density of the point cloud data does not reach the set density, the processor (210) can generate new points based on the i+1th frame for the point cloud data containing new points based on the i-th frame.

[0109] The set density is set according to the data collection frame rate of consecutive frames, and the higher the data collection frame rate, the lower the set density can be set. This is because a high data collection frame rate means that point cloud data is updated more frequently, and in such cases, even if the density of the point cloud for each frame is low, the accuracy of the information can be compensated for.

[0110] Finally, in step S500, the processor (210) can perform object recognition based on the second point cloud data.

[0111] That is, in one embodiment, by applying a point cloud augmentation technique of a specific object in the current frame through prediction, it is possible to compensate for the problem of insufficient number of measured point cloud data compared to LiDAR, which is a disadvantage of 4D imaging radar.

[0112] The embodiments according to the present disclosure described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. In this case, the medium may include a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a hardware device specifically configured to store and execute program instructions, such as a ROM, RAM, or flash memory.

[0113] Meanwhile, the above computer program may be one specifically designed and configured for the present disclosure, or one known and available to a person skilled in the art of computer software. Examples of computer programs may include machine code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0114] In the specification of this disclosure (particularly in the claims), the use of the term "above" and similar descriptive terms may be in both singular and plural. Furthermore, where a range is described in this disclosure, it is to include an invention to which individual values ​​belonging to said range are applied (unless otherwise stated), as is equivalent to describing each individual value constituting said range in the detailed description of the invention.

[0115] Unless explicitly stated otherwise, the steps constituting the method according to the present disclosure may be performed in a suitable order. The present disclosure is not necessarily limited by the order in which the steps are described. The use of any examples or exemplary terms (e.g., etc.) in the present disclosure is merely for the purpose of describing the present disclosure in detail and, unless limited by the claims, the scope of the present disclosure is not limited by such examples or exemplary terms. Furthermore, a person skilled in the art will understand that various modifications, combinations, and changes may be made according to design conditions and factors within the scope of the claims or equivalents to which they are added.

[0116] Accordingly, the scope of the present disclosure should not be limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the scope of the present disclosure. Explanation of the symbols

[0117] 1 : Object Recognition System 10: Autonomous Vehicles 100: Imaging Radar 200 : Object recognition device 210 : Processor 220 : Memory 230 : Display 300 : Server 400 : Network

Claims

Claim 1 An object recognition method performed by an object recognition device, comprising: receiving first point cloud data of the i-1th frame among consecutive frames, wherein i is a natural number greater than or equal to 2; outputting information about at least one object by extracting features from the first point cloud data; performing tracking of the at least one object in the i-th frame based on the information about the at least one object to predict a point cloud according to the position of the at least one object in the i-th frame; merging the point cloud data according to the position of the at least one object in the i-th frame with the first point cloud data to generate enhanced second point cloud data; and performing object recognition based on the second point cloud data, wherein the step of generating the second point cloud data comprises: generating new points based on the i-th frame for all points of the first point cloud, and determining whether the density of the point cloud data containing the new points based on the i-th frame has reached a set density. An object recognition method comprising the step of generating a new point based on the i+1th frame for point cloud data containing new points based on the i-th frame when the density of the point cloud data does not reach the set density. Claim 2 In claim 1, the step of predicting a point cloud according to a position in the i-th frame comprises: a step of predicting the direction of movement of the at least one object in the i-th frame based on a relative velocity value and a position value for the at least one object in the i-1-th frame; and a step of predicting the position value of the point cloud data in the i-th frame for the at least one object according to the predicted direction of movement, an object recognition method. Claim 3 An object recognition method according to claim 2, further comprising the step of changing the data collection frame rate based on the relative speed of at least one object in the i-1st frame, wherein the step of changing the data collection frame rate comprises changing the data collection frame rate to be longer than a set value if the relative speed of at least one object in the i-1st frame is less than a reference value, and changing the data collection frame rate to be shorter than a set value if the relative speed is greater than or equal to the reference value. Claim 4 In claim 2, the object recognition method further comprises the step of calculating accuracy based on the result of comparing a predicted position value and a measured position value for point cloud data in the i-th frame for at least one object, and the step of generating enhanced second point cloud data comprises the step of adjusting the ratio of merging the predicted position value of the point cloud data in the i-th frame for at least one object with the first point cloud data based on the accuracy. Claim 5 An object recognition method according to claim 2, further comprising the step of determining whether there is a region of interest with respect to the point cloud data in the i-th frame for the at least one object, and the step of generating the enhanced second point cloud data comprising the step of adjusting the ratio of merging the location value of the predicted point cloud data in the i-th frame for the at least one object into the first point cloud data based on whether there is a region of interest. Claim 6 In claim 1, the first point cloud data and the second point cloud data are data acquired by a 4D imaging radar, and the object recognition method includes information regarding the position, movement speed, movement direction, and height of an object detected by the 4D imaging radar. Claim 7 An object recognition method according to claim 6, further comprising the step of embedding first point cloud data acquired from the 4D imaging radar into a 5-dimensional vector space after the receiving step and before the output step. Claim 8 delete Claim 9 An object recognition method according to claim 1, wherein the set density is set according to the data collection frame rate of the consecutive frames, and the higher the data collection frame rate, the lower the set density is set. Claim 10 A computer-readable recording medium having a program recorded thereon that enables the method of any one of claims 1 through 7 and 9. Claim 11 As an object recognition device, memory; and includes at least one processor connected to the memory and configured to execute computer-readable instructions contained in the memory, wherein the at least one processor is configured to perform the operation of receiving first point cloud data of the i-1th frame among consecutive frames (i is a natural number greater than or equal to 2), the operation of outputting information about at least one object by extracting features from the first point cloud data, the operation of predicting a point cloud according to the position of the at least one object in the i-th frame by performing tracking of the at least one object in the i-th frame based on the information about the at least one object, the operation of generating enhanced second point cloud data by merging the point cloud data according to the position of the at least one object in the i-th frame with the first point cloud data, and the operation of performing object recognition based on the second point cloud data, wherein the operation of generating the second point cloud data comprises the step of generating new points based on the i-th frame for all points of the first point cloud, and the operation of determining whether the density of the point cloud data containing the new points based on the i-th frame has reached a set density, and the density of the point cloud data is the An object recognition device comprising the operation of generating a new point based on the i+1th frame for point cloud data containing new points based on the i-th frame when a set density is not reached. Claim 12 An object recognition device according to claim 11, wherein the operation of predicting a point cloud according to a position in the i-th frame comprises: an operation of predicting the direction of movement of the at least one object in the i-th frame based on a relative velocity value and a position value for the at least one object in the i-1-th frame; and an operation of predicting the position value of the point cloud data in the i-th frame for the at least one object according to the predicted direction of movement. Claim 13 In claim 12, the at least one processor is configured to further perform an operation of changing the data collection frame rate based on the relative speed of the at least one object in the i-1st frame, and the operation of changing the data collection frame rate includes changing the data collection frame rate to be longer than a set value if the relative speed of the at least one object in the i-1st frame is less than a reference value, and changing the data collection frame rate to be shorter than a set value if the relative speed is greater than or equal to the reference value, an object recognition device. Claim 14 In claim 12, the at least one processor is configured to further perform an operation of calculating accuracy based on a comparison result between a predicted position value and a measured position value for point cloud data in the i-th frame for the at least one object, and the operation of generating enhanced second point cloud data includes an operation of adjusting the ratio of merging the predicted position value of the point cloud data in the i-th frame for the at least one object into the first point cloud data based on the accuracy. Claim 15 In claim 12, the at least one processor is configured to further perform an operation of determining whether there is a region of interest with respect to the point cloud data in the i-th frame for the at least one object, and the operation of generating the enhanced second point cloud data includes an operation of adjusting the ratio of merging the location value of the point cloud data in the i-th frame for the predicted at least one object into the first point cloud data based on whether there is a region of interest, an object recognition device. Claim 16 In claim 11, the first point cloud data and the second point cloud data are data acquired by a 4D imaging radar, and the object recognition device includes information regarding the position, movement speed, movement direction, and height of an object detected by the 4D imaging radar. Claim 17 An object recognition device according to claim 16, wherein the at least one processor is configured to further perform an operation of embedding first point cloud data acquired from the 4D imaging radar into a 5-dimensional vector space after the receiving operation and before the output operation. Claim 18 delete Claim 19 An object recognition device according to claim 11, wherein the setting density is set according to the data collection frame rate of the consecutive frames, and the higher the data collection frame rate, the lower the setting density is set.

Images