Apparatus and method for detecting objects

The object detection method addresses incomplete point cloud data issues by supplementing and enhancing point cloud features using deep learning networks, improving object detection accuracy and interaction in augmented reality.

KR103046791B1Active Publication Date: 2026-11-11SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
KR1020210088389
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-11-26
Filing Date
2021-07-06
Publication Date
2026-11-11
Estimated Expiration
2041-07-06

AI Technical Summary

Technical Problem

Existing augmented reality technology faces challenges in performing effective and accurate object detection and recognition due to limitations in depth sensor operating principles, particularly with incomplete point cloud data from occlusions, reflections, and transparency.

Method used

An object detection method that includes point cloud supplementation, high-dimensional feature extraction, and geometric and appearance feature analysis using deep learning networks to enhance point cloud data and improve object detection accuracy.

Benefits of technology

Enhances object detection accuracy by filling data gaps in point clouds, improving robustness in detecting occluded and small objects, and enabling realistic interaction between real and virtual objects in augmented reality systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112021077787160-PAT00007_ABST
    Figure 112021077787160-PAT00007_ABST
Patent Text Reader

Abstract

An object detection device and method are provided. The method acquires a first point cloud feature based on point cloud data of an image and determines at least one object in the image based on the first point cloud feature.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The following embodiments relate to the field of image processing, and in particular to an object detection method and apparatus. Background Technology

[0003] Augmented reality technology adds virtual content to real scenes in front of the user to provide the user with a real information experience. In three-dimensional space, an augmented reality system requires high-precision real-time processing and understanding of the 3D state of surrounding objects to present high-quality virtual and real fusion effects to the user.

[0004] However, existing augmented reality technology is limited by the operating principles of depth sensors during object detection and recognition, making it difficult to perform effective and accurate object detection and recognition. The problem to be solved

[0007] An object detection method according to one embodiment of the present disclosure includes the step of acquiring a first point cloud feature based on point cloud data of an image; and the step of determining at least one object in the image based on the first point cloud feature.

[0008] At this time, the object detection method further includes a step of performing point cloud supplementation on the point cloud data of the image prior to the step of acquiring the first point cloud feature, and the step of acquiring the first point cloud feature can acquire the first point cloud feature based on the supplemented point cloud data.

[0009] At this time, the step of performing point cloud supplementation on the point cloud data of the image may include: a step of extracting high-dimensional features of the point cloud data of the image; and a step of supplementing the point cloud based on the high-dimensional features to obtain the supplemented point cloud data.

[0010] At this time, the step of determining at least one object in the image may include: a step of obtaining a first appearance feature and a first geometric feature of each object in the image based on the first point cloud feature; a step of obtaining a relationship weight between candidate regions corresponding to each object based on the first appearance feature and the first geometric feature of each object in the image; a step of obtaining a second appearance feature and a second geometric feature of each object in the image based on the first point cloud feature and the relationship weight; and a step of determining the at least one object in the image based on the second appearance feature and the second geometric feature of each object in the image.

[0011] At this time, the step of obtaining a second appearance feature and a second geometric feature of each object in the image may include: a step of obtaining a geometric weight between candidate regions based on the first geometric feature of each object in the image; a step of obtaining an appearance weight between candidate regions based on the first appearance feature of each object in the image; a step of obtaining a relationship weight between candidate regions based on the geometric weight and the appearance weight; a step of obtaining a second point cloud feature based on the first point cloud feature and the relationship weight; and a step of obtaining the second appearance feature and the second geometric feature of each object in the image based on the second point cloud feature.

[0012] At this time, the step of obtaining geometric weights between the candidate regions may include: a step of obtaining a geometric feature-related vector between the candidate regions based on the first geometric feature of each object of the image; and a step of obtaining the geometric weights between the candidate regions based on the geometric feature-related vector between the candidate regions.

[0013] At this time, the first geometric feature, the second geometric feature, and the vector related to the geometric feature may be related to at least one of the center position, size, and direction of the object.

[0014] An object detection device according to one embodiment of the present disclosure includes: a feature acquisition unit configured to acquire a first point cloud feature based on point cloud data of an image; and an object determination unit configured to determine at least one object in the image based on the first point cloud feature.

[0015] At this time, the object detection device further includes a supplementation unit that performs point cloud supplementation on the point cloud data of the image, and the feature acquisition unit can acquire the first point cloud feature based on the supplemented point cloud data.

[0016] At this time, the supplementary unit can extract high-dimensional features of the point cloud data of the image and supplement the point cloud based on the high-dimensional features to obtain the supplemented point cloud data.

[0017] At this time, the object determination unit may obtain a first appearance feature and a first geometric feature of each object in the image based on the first point cloud feature, obtain a relationship weight between candidate regions corresponding to each object based on the first appearance feature and the first geometric feature of each object in the image, obtain a second appearance feature and a second geometric feature of each object in the image based on the first point cloud feature and the relationship weight, and determine at least one object in the image based on the second appearance feature and the second geometric feature of each object in the image.

[0018] At this time, when the object determination unit obtains the second appearance feature and the second geometric feature of each object in the image, it may obtain a geometric weight between the candidate regions based on the first geometric feature of each object in the image, obtain an appearance weight between the candidate regions based on the first appearance feature of each object in the image, obtain a relationship weight between the candidate regions based on the geometric weight and the appearance weight, obtain a second point cloud feature based on the first point cloud feature and the relationship weight, and obtain the second appearance feature and the second geometric feature of each object in the image based on the second point cloud feature.

[0019] At this time, when the object determination unit obtains geometric weights between the candidate regions, it may obtain a geometric feature-related vector between the candidate regions based on the first geometric feature of each object of the image, and obtain geometric weights between the candidate regions based on the geometric feature-related vector between the candidate regions.

[0020] At this time, the first geometric feature, the second geometric feature, and the vector related to the geometric feature may be related to at least one of the center position, size, and direction of the object.

[0021] An electrical device according to one embodiment of the present disclosure includes: a memory for storing an executable command; and a processor for acquiring a first point cloud feature based on point cloud data of an image according to the command, and determining at least one object in the image based on the first point cloud feature.

[0022] At this time, the processor may perform point cloud supplementation on the point cloud data of the image before acquiring the first point cloud feature, and acquire the first point cloud feature based on the supplemented point cloud data. Brief explanation of the drawing

[0024] FIG. 1 is a diagram illustrating an example of a method for detecting an object according to one embodiment. FIG. 2 is a flowchart illustrating a schematic process of detecting an object according to one embodiment. FIG. 3 is a diagram illustrating an example of supplementing point cloud data according to one embodiment. FIG. 4 is a flowchart illustrating the process of supplementing point cloud data according to one embodiment. FIG. 5 is a diagram illustrating an example of the effect of supplementing point cloud data according to one embodiment. FIG. 6 is a flowchart illustrating the process of determining an object according to one embodiment. FIG. 7 is a diagram illustrating the process of acquiring and using the external features and geometric features of an object according to one embodiment. FIG. 8 is a flowchart illustrating the process of obtaining external features and geometric features of an object according to one embodiment. FIG. 9 is a diagram illustrating the schematic configuration of an object detection device according to one embodiment. FIG. 10 is a diagram illustrating the schematic configuration of an electrical device for detecting an object according to one embodiment. Specific details for implementing the invention

[0025] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, various modifications may be made to the embodiments, and thus the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, and substitutions to the embodiments are included within the scope of the rights.

[0026] The terms used in the embodiments are for illustrative purposes only and should not be interpreted as intended to be limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0027] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the embodiments pertain. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0028] In addition, when describing with reference to the attached drawings, identical components are assigned the same reference numeral regardless of drawing symbols, and redundant descriptions thereof are omitted. In describing the embodiments, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the embodiments, such detailed description is omitted.

[0029] In addition, terms such as first, second, A, B, (a), (b), etc., may be used when describing the components of the embodiments. These terms are intended only to distinguish the components from other components, and the nature, order, or sequence of the components is not limited by the terms. Where it is stated that a component is "connected," "combined," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that another component may also be "connected," "combined," or "connected" between each component.

[0030] Components included in any one embodiment and components having common functions shall be described using the same names in other embodiments. Unless otherwise stated, the description in any one embodiment may also apply to other embodiments, and specific descriptions shall be omitted to the extent of overlap.

[0031] Hereinafter, an object detection device and method according to an embodiment of the present invention will be described in detail with reference to the attached FIGS. 1 to 10.

[0032] FIG. 1 is a diagram illustrating an example of a method for detecting an object according to one embodiment.

[0033] Referring to FIG. 1, when an object detection device receives point cloud data (101) of an image, it performs point cloud supplementation (110), obtains a first point cloud feature based on the supplemented point cloud data (120), confirms object relationships based on relevant information related to the object (130), and can determine at least one object in the image (140).

[0034] The specific process for performing point cloud supplementation (110) is as follows.

[0035] The object detection device uses a coding network (111) to extract point cloud features (112), which are high-dimensional features of the point cloud data (101) of an image.

[0036] Then, the object detection device uses a decoding network (113) to obtain point cloud data (114) that is supplemented with point cloud based on point cloud features (112).

[0037] The specific process for obtaining the first point cloud feature (120) is as follows.

[0038] The object detection device can obtain downsampled point cloud features (122) by performing PointNet++ feature extraction (121) on the supplemented point cloud data (114). Then, the object detection device can obtain a point cloud feature (124) corresponding to the geometric center of the object after clustering, i.e., a first point cloud feature, by performing the Hough Voting algorithm (123) on the downsampled point cloud features (122).

[0039] The specific process for verifying object relationships (130) is as follows.

[0040] The object detection device can obtain the first appearance feature (132) and the first geometric feature (133) of each object in an image based on the first point cloud feature (124) using the first area candidate network (131).

[0041] The object detection device obtains a related information feature (134) based on the first appearance feature (132) and the first geometric feature (133).

[0042] The object detection device can obtain second appearance features and second geometric features of each object in an image based on first point cloud features and relationship information features (134) using a second area candidate network (135).

[0043] The specific process of object determination (140) is as follows.

[0044] The object detection device can determine at least one object in an image based on the second appearance features and second geometric features of each object in the image using the second area candidate network (135) (140).

[0046] FIG. 2 is a flowchart illustrating a schematic process of detecting an object according to one embodiment.

[0047] Referring to FIG. 2, the object detection device can acquire a first point cloud feature based on the point cloud data of an image (210).

[0048] More specifically, the object detection device can obtain downsampled point cloud features (122) by performing PointNet++ feature extraction (121) on the point cloud data of an image as in the example of FIG. 1 described above. Then, the object detection device can obtain a point cloud feature (124) corresponding to the geometric center of an object after clustering, i.e., a first point cloud feature, by performing the Hough Voting algorithm (123) on the downsampled point cloud features (122). The PointNet++ feature extraction and Hough Voting algorithm used here are merely examples and are not limitations, and a person of ordinary knowledge can obtain the first point cloud feature using other algorithms.

[0049] The object detection device can determine at least one object in an image based on the first point cloud feature (220).

[0050] In addition, before acquiring the first point cloud feature, point cloud enhancement can be performed on the point cloud data of the image, and in step 220, clustered point cloud features can be acquired based on the point cloud enhanced point cloud data.

[0051] Hereinafter, the point cloud supplementation step will be described in more detail with reference to FIGS. 3 to 5, and the step S220 will be described in more detail with reference to FIG. 6.

[0052] Point cloud data acquired from depth sensors often contains holes because there is a lot of missing data due to occlusion, surface reflections, transparency, and other objects. Since these incomplete point clouds have a significant impact on 3D object detection performance, the present disclosure may add point cloud supplementation work.

[0054] FIG. 3 is a diagram illustrating an example of supplementing point cloud data according to one embodiment.

[0055] FIG. 4 is a flowchart illustrating the process of supplementing point cloud data according to one embodiment.

[0056] FIG. 5 is a diagram illustrating an example of the effect of supplementing point cloud data according to one embodiment.

[0057] Referring to FIGS. 3 and 4, the object detection device can extract high-dimensional features (311) of point cloud data (301) of an image (410).

[0058] Step 410 can be performed by a coding network (310). Here, the coding network (310) is merely an example and not a limitation, and may be a deep learning network based on convolutional neurons. The input of the coding network (310) is the original point cloud data (301), and the output is a high-dimensional feature (311) extracted from the coding network, which can describe the shape and boundaries of an object. Additionally, the coding network (310) may have various forms, and the extracted high-dimensional feature may be at least one of semantic features of object categories, contour features of objects, and depth information features, or a combination of these features. The commonality is that, in a situation where the actual data is not altered, even if there are omissions in the point cloud data, the extracted feature may not be affected by the missing data or may be affected less, and is essentially not altered.

[0059] The object detection device can obtain supplemented point cloud data (321) by supplementing point cloud data based on high-dimensional features (311) (420).

[0060] Step 420 can be performed by a decoding network (320), and the augmented point cloud data (321) has the same size as the point cloud data (301). Here, the decoding network (320) is an example and not a limitation and may be a deep learning network based on convolutional neurons. The decoding network (320) uses a feature map (composed of high-level features) (311) output from the coding network (310) as input, and the output is augmented and dense point cloud data (321).

[0061] Referring to FIG. 5, 510 illustrates an image of an actual scene, 520 illustrates point cloud data with a large number of holes, and 530 illustrates point cloud data with holes supplemented in the point cloud data. Regarding data omissions in the point cloud data, higher quality point cloud data can be obtained by filling in the missing holes in the point cloud through a neural network model with a CODEC (coder and decoder) network structure.

[0063] FIG. 6 is a flowchart illustrating the process of determining an object according to one embodiment.

[0064] Referring to FIG. 6, the object detection device can acquire a first appearance feature and a first geometric feature of each object in an image based on a first point cloud feature (610). Here, the geometric feature is merely an example and not a limitation and may be related to at least one of the center position, size, and orientation of the object.

[0065] And, the object detection device can obtain a relationship weight between two candidate regions based on the first appearance feature and the first geometric feature of each object in the image (620). At this time, the candidate region corresponds to an object.

[0066] And, the object detection device can acquire second appearance features and second geometric features of each object in the image based on first point cloud features and relationship weights (630).

[0067] And, the object detection device can determine at least one object in an image based on the second appearance feature and second geometric feature of each object in the image (640). As a more specific example, the category, center position, size, direction, etc. of an object can be determined by inputting the appearance feature and geometric feature of the object into a predetermined neural network. At this time, the predetermined neural network may be a deep learning network having a Multi-layer Perceptron (MLP) structure.

[0068] It is difficult to effectively and accurately detect and recognize objects when there are occlusions, cutoffs in the field of view, or small objects.

[0069] Multiple objects have the characteristic that there is a certain coexistence relationship between objects (e.g., a mouse is generally next to a keyboard), a certain relationship in size (e.g., several chairs of the same size are generally next to a table), and a fixed pattern and habit in orientation (e.g., a bedside table and a bed are placed parallel). In other words, the degree of association between objects is strongly correlated with the object's category, location, size, and orientation.

[0070] After clustering point cloud features, in step 640, which determines the object illustrated in FIG. 6, considering other object features related to the current object in the environment together and using point cloud features as input to predict category, geometry, and size helps improve the accuracy of determining the current object. Below, steps 610 through 630 will be described in detail with reference to FIG. 7 and 8.

[0072] FIG. 7 is a diagram illustrating the process of acquiring and using the external features and geometric features of an object according to one embodiment.

[0073] FIG. 8 is a flowchart illustrating the process of obtaining external features and geometric features of an object according to one embodiment.

[0074] Referring to FIGS. 7 and 8, the object detection device can obtain a first appearance feature (731) and a first geometric feature (732) of each object in an image based on a first point cloud feature (710) (810). At this time, step 810 can be performed by a first region candidate network (720). As a more specific example, the first point cloud feature (710), which has no clear physical meaning, passes through the first region candidate network (720), and each candidate region corresponding to an object can obtain four different features (first geometric features (732)) corresponding to the appearance (first appearance feature (731)), center position (x, y, z), size (w (width), h (height), d (length)), and direction (roll, pitch, yaw) of each candidate region object. At this time, the center position (x, y, z), size (w, h, d), and direction (roll, pitch, yaw) of the object correspond to the first geometric feature (732). Here, as an example, the first region candidate network (720) may be a deep learning network based on convolutional neurons or a deep learning network of an MLP structure.

[0075] And, the object detection device can obtain a geometric weight between two candidate regions based on a first geometric feature (732) of each object in the image (820). More specifically, based on the first geometric feature (720) of each object in the image, a vector related to the geometric feature between each of the two candidate regions can be obtained, and based on the vector related to the geometric feature between each of the two candidate regions, a geometric weight (741) between each of the two candidate regions can be obtained. The following <Equation 1> shows a vector (εG) related to the geometric feature (fGm, fGn) between two candidate regions (m, n).

[0076] [Mathematical Formula 1]

[0077]

[0078] Here, x, y, z correspond to the center position features of the object output by the first region candidate network, w, h, d correspond to the size features of the object output by the first region candidate network, roll, pitch, yaw correspond to the orientation features of the object output by the first region candidate network, x, y, z, w, h, d, roll, pitch, yaw and the characters m, n written below represent x, y, z, w, h, d, roll, pitch, yaw of candidate region m or n, and ε G is a 6-dimensional vector related to the center position, size, and orientation of the object.

[0079] More specifically, step 820, which obtains geometric weights between candidate regions based on geometric feature-related vectors between candidate regions, is a 6-dimensional vector (ε G ) neural network(W G By inputting into ) and mapping to a new feature space, the vector (f) of two candidate regions in the high-dimensional space (i.e., the aforementioned new feature space) G m , f G nThe geometric weight (W) between two candidate regions (m, n) for the distance ) G mn It may include a step of calculating with ). Here, the neural network (W G ) can be a deep learning network with an MLP structure. In addition, the geometric weights (W) obtained through the following <Equation 2> G mn It can take a positive value for ).

[0080] [Mathematical Formula 2]

[0081]

[0082] And, the object detection device can obtain an appearance weight (741) between two candidate regions based on the first appearance feature (731) of each object in the image (830). More specifically, the appearance feature (f) of two candidate regions (m, n) A m , f A n ) neural network(W K , W Q After inputting into ) to map to a new feature space, the appearance features (W) between two candidate regions (m, n) are obtained through <Equation 3> below. A mn ) can be calculated. Here, the neural network (W K , W Q ) can be a deep learning network with an MLP structure.

[0083] [Mathematical Formula 3]

[0084]

[0085] At this time, d k can be a predetermined constant representing a predetermined multiple of the number of candidate regions.

[0086] And, the object detection device has geometric weights (W G mn )(742) and appearance weight (W A mnBased on )(741), the relationship weight (743) between two candidate regions can be obtained (840). More specifically, the relationship weight (W) between candidate regions can be obtained through <Equation 4> below. mn ) can be obtained.

[0087] [Mathematical Formula 4]

[0088]

[0089] In this case, k can represent natural numbers from 1 to K, and K can be the number of candidate regions.

[0090] And, the object detection device can obtain a second point cloud feature (750) based on the first point cloud feature (710) and the relationship weight (743) (850). At this time, the second point cloud feature (750) may include relevant information. More specifically, the object detection device obtains the first point cloud feature (f) through the following <Equation 5> m After inputting ) into the neural network (Wv), the second point clout feature (f) is obtained by weighted summation. R (n)) can be obtained. The second point cloud feature (750) has the same quantity and feature dimension as the first point cloud feature (710). Here, the neural network (Wv) may be a deep learning network with an MLP structure.

[0091] [Mathematical Formula 5]

[0092]

[0093] In this case, m can represent a natural number from 1 to K, and K can be the number of candidate regions.

[0094] And, the object detection device can acquire a second appearance feature (771) and a second geometric feature (772) of each object in an image based on a second point cloud feature (750) (860). At this time, step 860 can be performed by a second region candidate network (760). Here, the second region candidate network (760) may be a deep learning network based on convolutional neurons or a deep learning network with an MLP structure, and the first region candidate network (720) and the second region candidate network (760) may use completely identical network structures and may share / unshare network weights.

[0095] The second external feature (771) and the second geometric feature (772) are subsequently used for object detection (780).

[0097] FIG. 9 is a diagram illustrating the schematic configuration of an object detection device according to one embodiment.

[0098] Referring to FIG. 9, the object detection device may include a supplementary unit (910), a feature acquisition unit (920), and an object determination unit (930).

[0099] The complementary unit (910) can extract point cloud features (112), which are high-dimensional features of the point cloud data (101) of the image, and obtain point cloud data (114) in which the point cloud is complemented based on the point cloud features (112). The complementary unit (910) may be optional and may not be included.

[0100] The feature acquisition unit (920) can acquire a first point cloud feature based on the point cloud data of the image or the supplemented point cloud data.

[0101] The object determination unit (930) can determine at least one object among the images based on the first point cloud feature.

[0102] Additionally, an object detection device according to one embodiment of the present disclosure may further include other configurations for implementing features corresponding to different steps of the object detection method of the present disclosure described above, and are not repeated further herein.

[0104] FIG. 10 is a diagram illustrating the schematic configuration of an electrical device for detecting an object according to one embodiment.

[0105] Referring to FIG. 10, the electric device (1000) may include a processor (1010), a memory (1020), an input unit (1030), and an output unit (1040).

[0106] The memory (1020) stores an operating system, an application program, and storage data for controlling the overall operation of the electric device (1000), and can store an executable command for object detection according to the present disclosure.

[0107] The input unit (1030) can receive point cloud data.

[0108] The output unit (1040) can output information regarding at least one object of the image detected by the processor (1010).

[0109] The processor (1010) may include a supplementary unit (1011), a feature acquisition unit (1012), and an object determination unit (1013).

[0110] The processor (1010) performs point cloud enhancement on the point cloud data of an image through the enhancement unit (1011) according to a command stored in memory (1020), acquires a first point cloud feature based on the enhanced point cloud data through the feature acquisition unit (1012), and can determine at least one object in the image based on the first point cloud feature through the object determination unit (1013). The operation of the enhancement unit (1011) is optional and can be omitted.

[0112] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0113] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0114] According to the present disclosure, an object detection method in an electronic device can obtain output data that recognizes an image or point cloud features within an image by using image data as input data for an artificial intelligence model.

[0115] Artificial intelligence models can be acquired through training. Here, 'acquisition through training' refers to acquiring a predefined operational rule or an artificial intelligence model configured to perform necessary features (or objectives) by training a base artificial intelligence model with multiple training data using a training algorithm.

[0116] An artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and performs neural network calculations through calculations between the calculation result of the previous layer and the multiple weight values.

[0117] Visual understanding is a technology that recognizes and processes objects, such as human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / positioning, or image enhancement.

[0118] According to one embodiment of the present disclosure, the lack of depth data due to object occlusion, reflection, transparency, etc., can be improved by supplementing point cloud data, and the data quality for object detection can be improved. By considering the relationships between objects, the accuracy and robustness of object detection or recognition performed in cases of object occlusion, cutoff of view, and small objects can be improved through the extraction and fusion of relationship information between multiple objects. Based on the above solution, 6-DOF geometric estimation of 3D objects based on point cloud data can be performed to improve the efficiency, accuracy, and robustness of the system in augmented reality applications. The object detection method and device can detect objects of interest around a user in an augmented reality system and a depth image and be used for 6-DOF geometric estimation. Based on the acquired detection results, virtual objects rendered in the augmented reality system can be controlled to enable realistic and natural interaction between real objects and virtual objects.

[0119] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0120] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A method for detecting an object in an object detection device, comprising: a step of obtaining a first point cloud feature based on point cloud data of an image; a step of obtaining a first appearance feature and a first geometric feature of each object in the image based on the first point cloud feature; a step of obtaining an appearance weight between candidate regions based on the first appearance feature of each object in the image; a step of obtaining a geometric weight between candidate regions based on the first geometric feature of each object in the image; a step of obtaining a second appearance feature and a second geometric feature of each object in the image based on the first point cloud feature, the appearance weight, and the geometric weight; and a step of determining at least one object in the image based on the second appearance feature and the second geometric feature of each object in the image. Claim 2 In claim 1, prior to the step of acquiring the first point cloud feature, the method further includes the step of performing point cloud supplementation on the point cloud data of the image, and the step of acquiring the first point cloud feature is a method of acquiring the first point cloud feature based on the supplemented point cloud data. Claim 3 In paragraph 2, the step of performing point cloud supplementation on the point cloud data of the image comprises: a step of extracting high-dimensional features of the point cloud data of the image; and a step of supplementing the point cloud based on the high-dimensional features to obtain the supplemented point cloud data. Claim 4 A method according to claim 1, further comprising the step of controlling the operation of an augmented reality system based on at least one determined object. Claim 5 In claim 1, the step of obtaining a second appearance feature of each object of the image comprises the step of applying a relationship weight corresponding to each object of the image to the first point cloud feature of each object of the image, wherein the relationship weight is based on the appearance weight and the geometric weight. Claim 6 A method according to claim 1, wherein the step of obtaining geometric weights between candidate regions comprises: a step of obtaining a geometric feature-related vector between candidate regions based on the first geometric feature of each object of the image; and a step of obtaining geometric weights between candidate regions based on the geometric feature-related vector between candidate regions. Claim 7 In claim 6, the first geometric feature, the second geometric feature, and the geometric feature-related vector are related to at least one of the center position, size, and direction of the object. Claim 8 An object detection device comprising: a feature acquisition unit configured to acquire a first point cloud feature based on point cloud data of an image; and an object determination unit configured to acquire an appearance weight between candidate regions based on a first appearance feature of each object in the image, acquire a geometric weight between candidate regions based on a first geometric feature of each object in the image, acquire a second appearance feature and a second geometric feature of each object in the image based on the first point cloud feature, the appearance weight, and the geometric weight, and determine at least one object in the image based on the second appearance feature and the second geometric feature of each object in the image. Claim 9 In claim 8, the object detection device further includes a supplementation unit that performs point cloud supplementation on the point cloud data of the image, and the feature acquisition unit acquires the first point cloud feature based on the supplemented point cloud data. Claim 10 In claim 9, the supplementary unit is an object detection device that extracts high-dimensional features of point cloud data of the image and supplements the point cloud based on the high-dimensional features to obtain the supplemented point cloud data. Claim 11 delete Claim 12 In claim 8, the object determination unit, when acquiring a second appearance feature of each object of the image, applies a relationship weight corresponding to each object of the image to the first point cloud feature of each object of the image, and the relationship weight is an object detection device based on the appearance weight and the geometric weight. Claim 13 In claim 8, the object determination unit, when obtaining geometric weights between candidate regions, obtains a geometric feature-related vector between candidate regions based on the first geometric feature of each object in the image, and obtains the geometric weights between candidate regions based on the geometric feature-related vector between candidate regions. Claim 14 In paragraph 13, the first geometric feature, the second geometric feature, and the geometric feature-related vector are an object detection device associated with at least one of the center position, size, and direction of an object. Claim 15 An electronic device for detecting objects comprises one or more processors including a processing circuit, and a memory storing instructions that cause the electronic device to operate when the one or more processors execute individually or collectively, wherein the instructions, when executed by the one or more processors, cause the electronic device to perform the steps of: obtaining a first point cloud feature based on point cloud data of an image; obtaining an appearance weight between candidate regions based on a first appearance feature of each object in the image; obtaining a geometric weight between candidate regions based on a first geometric feature of each object in the image; obtaining a second appearance feature and a second geometric feature of each object in the image based on the first point cloud feature, the appearance weight, and the geometric weight; and determining at least one object in the image based on the second appearance feature and the second geometric feature of each object in the image. Claim 16 An electronic device according to claim 15, wherein the processor performs point cloud supplementation on the point cloud data of the image before acquiring the first point cloud feature, and acquires the first point cloud feature based on the supplemented point cloud data. Claim 17 A computer-readable recording medium characterized by having a program recorded thereon for executing the method of any one of claims 1 to 7.