Object detection method and apparatus

Abstract

Provided is an object detection method, comprising: obtaining first point cloud features based on point cloud data of an image; and determining at least one object in the image based on the first point cloud features. Meanwhile, the above object detection method can be performed using an artificial intelligence model.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to an object detection method and apparatus. Background Technology

[0002] Augmented reality (AR) technology provides users with a realistic information experience by adding virtual content to the real-world scene in front of them. In three-dimensional space, AR systems need to have high-precision real-time processing and understanding of the three-dimensional state of surrounding objects to present a high-quality virtual-real fusion effect to the user.

[0003] However, existing augmented reality technologies are limited by the working principle of depth sensors, making it difficult to perform object detection and recognition effectively and accurately. Summary of the Invention

[0004] According to an exemplary embodiment of the present invention, an object detection method is provided, comprising: acquiring first point cloud features based on point cloud data of an image; and determining at least one object in the image based on the first point cloud features.

[0005] Before acquiring the first point cloud features, point cloud completion can be performed on the point cloud data of the image.

[0006] The steps for point cloud completion of image point cloud data may include: extracting high-dimensional features from the image point cloud data; and obtaining point cloud data for point cloud completion based on the high-dimensional features.

[0007] The step of determining at least one object in an image may include: obtaining a first appearance feature and a first geometric feature for each object in the image based on a first point cloud feature; obtaining a relationship weight between every two candidate regions based on the first appearance feature and the first geometric feature for each object in the image, wherein the candidate regions correspond to objects; obtaining a second appearance feature and a second geometric feature for each object in the image based on the first point cloud feature and the relationship weight; and determining the at least one object in the image based on the second appearance feature and the second geometric feature for each object in the image.

[0008] The steps of obtaining the second appearance feature and the second geometric feature of each object in the image may include: obtaining the geometric weight between every two candidate regions based on the first geometric feature of each object in the image; obtaining the appearance weight between every two candidate regions based on the first appearance feature of each object in the image; obtaining the relationship weight between every two candidate regions based on the geometric weight and the appearance weight; obtaining the second point cloud feature based on the first point cloud feature and the relationship weight; and obtaining the second appearance feature and the second geometric feature of each object in the image based on the second point cloud feature.

[0009] The steps for obtaining the geometric weights between every two candidate regions may include: obtaining a geometric feature association vector between every two candidate regions based on the first geometric feature of each object in the image; and obtaining the geometric weights between every two candidate regions based on the geometric feature association vector between every two candidate regions.

[0010] Geometric features and their associated vectors can be related to at least one of the object’s center position, size, and orientation.

[0011] According to an exemplary embodiment of the present invention, an object detection apparatus is provided, comprising: a feature acquisition unit for acquiring first point cloud features based on point cloud data; and an object determination unit for determining at least one object in an image based on the first point cloud features.

[0012] According to an exemplary embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores computer-executable instructions, and when the instructions are executed by the processor, the aforementioned method is performed.

[0013] According to an exemplary embodiment of the present invention, a computer-readable medium is provided having computer-executable instructions stored thereon, which, when executed, perform the aforementioned method.

[0014] Further aspects and / or advantages of the general concept of the invention will be set forth in part in the description which follows, and in part will be obvious from the description or may be learned by practice of the general concept of the invention. Attached Figure Description

[0015] The above and other objects and features of the present invention will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0016] Figure 1 This is a schematic diagram illustrating an object detection method according to an exemplary embodiment of the present invention;

[0017] Figure 2 This is a flowchart illustrating an object detection method according to an exemplary embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram illustrating point cloud completion according to an exemplary embodiment of the present invention;

[0019] Figure 4 This is a flowchart illustrating point cloud completion according to an exemplary embodiment of the present invention;

[0020] Figure 5 This is a schematic diagram illustrating the point cloud completion effect according to an exemplary embodiment of the present invention;

[0021] Figure 6This is a flowchart illustrating the determination of an object according to an exemplary embodiment of the present invention;

[0022] Figure 7 This is a schematic diagram illustrating the acquisition of the appearance and geometric features of an object according to an exemplary embodiment of the present invention;

[0023] Figure 8 This is a flowchart illustrating the acquisition of the appearance and geometric features of an object according to an exemplary embodiment of the present invention;

[0024] Figure 9 This is a block diagram illustrating an object detection apparatus according to an exemplary embodiment of the present invention. Detailed Implementation

[0025] The present invention will now be described in detail with reference to embodiments thereof, examples of which are illustrated in the accompanying drawings, wherein the same reference numerals refer to the same parts throughout. The embodiments will be described below with reference to the accompanying drawings in order to explain the present invention.

[0026] Figure 1 This is a schematic diagram illustrating an object detection method according to an exemplary embodiment of the present invention. Figure 2 This is a flowchart illustrating an object detection method according to an exemplary embodiment of the present invention.

[0027] Reference Figure 1 and Figure 2 In step S210, the first point cloud features can be obtained based on the point cloud data of the image.

[0028] More specifically, PointNet++ feature extraction can be performed on the point cloud data of the image to obtain downsampled point cloud features, and the Hough Voting algorithm can be applied to the downsampled point cloud features to obtain the point cloud features corresponding to the geometric centers of the objects after clustering, i.e., the first point cloud features. It should be understood that the PointNet++ feature extraction and Hough Voting algorithm used here are only examples and not limitations, and those skilled in the art can certainly use other algorithms to obtain the first point cloud features.

[0029] In step S220, at least one object in the image can be determined based on the first point cloud features.

[0030] In addition, before obtaining the first point cloud features, the point cloud data of the image can be completed, and in step S220, the clustered point cloud features can be obtained based on the point cloud data completed.

[0031] The following will refer to the appendix. Figures 3-5 The steps for point cloud completion are described in more detail, and references will be provided in the appendix. Figure 6 Step S220 will be described in more detail.

[0032] Point cloud data acquired from depth sensors often contains holes due to occlusion, surface reflection, transparency, and other objects, resulting in a large amount of missing data. Such incomplete point clouds will greatly affect the performance of 3D object detection. Therefore, this invention incorporates a point cloud completion operation.

[0033] Figure 3 This is a schematic diagram illustrating point cloud completion according to an exemplary embodiment of the present invention. Figure 4 This is a flowchart illustrating point cloud completion according to an exemplary embodiment of the present invention. Figure 5 This is a schematic diagram illustrating the point cloud completion effect according to an exemplary embodiment of the present invention.

[0034] Reference Figure 3 and Figure 4 In step S410, high-dimensional features of the point cloud data of the image can be extracted, which can be performed by an encoding network. Here, as an example and not a limitation, the encoding network can be a deep learning network based on convolutional neurons. The input to the encoding network is the original point cloud data, and the output is the high-dimensional features extracted by the encoding network, which can describe the shape and boundaries of the object. Furthermore, the encoding network can take various forms, and the extracted high-dimensional features can be at least one of the semantic features of the object category, the contour features of the object, and depth information features, or a combination of these features. The common point is that, even if there are missing data in the point cloud data, the extracted features are largely unaffected by the missing data and remain essentially unchanged, provided the real data remains unchanged.

[0035] In step S420, point cloud data for point cloud completion can be obtained based on high-dimensional features. This step can be performed by a decoding network, and the point cloud data for point cloud completion has the same size as the point cloud data. Here, as an example and not a limitation, the decoding network can be a deep learning network based on convolutional neurons. The decoding network uses the feature map (composed of high-dimensional features) output by the encoding network as input, and the output is the completed and dense point cloud data.

[0036] Reference Figure 5 (a) shows an image of the actual scene, (b) shows point cloud data with numerous holes, and (c) shows point cloud data after completion. For missing data in point cloud data, a neural network model with an encoding / decoding network structure can be used to fill in the missing holes, thereby obtaining higher quality point cloud data.

[0037] Figure 6 This is a flowchart illustrating the determination of an object according to an exemplary embodiment of the present invention.

[0038] Reference Figure 6In step S610, based on the first point cloud features, a first appearance feature and a first geometric feature of each object in the image can be obtained. Here, by way of example only and not limitation, the geometric feature may be related to at least one of the object's center position, size, and orientation.

[0039] In step S620, the relationship weight between every two candidate regions can be obtained based on the first appearance feature and the first geometric feature of each object in the image, wherein the candidate region corresponds to the object.

[0040] In step S630, the second appearance feature and the second geometric feature of each object in the image can be obtained based on the first point cloud features and the relation weights.

[0041] In step S640, at least one object in the image is determined based on the second appearance feature and second geometric feature of each object in the image. More specifically, by way of example and not limitation, the object's category, center position, size, orientation, etc., can be determined by inputting the object's appearance feature and geometric feature into a predetermined neural network. Here, by way of example and not limitation, the predetermined neural network can be a deep learning network with a multilayer perceptron (MLP) structure.

[0042] Because object detection and recognition are difficult to perform effectively and accurately when objects are occluded, have obstructed views, or are small, and considering that multiple objects often have certain coexistence relationships (e.g., a mouse is usually next to a keyboard), size correlations (e.g., multiple chairs are usually the same size next to a table), and orientations that often conform to fixed patterns and habits (e.g., bedside tables and beds are placed parallel to each other), meaning that the degree of correlation between objects is strongly correlated with the object's category, position, size, and orientation, clustering point cloud features is necessary to further improve object detection and recognition. Figure 6 The steps for identifying objects shown consider features of other objects in the environment that are related to the current object. Using point cloud features as input and predicting category, geometry, and size helps improve the accuracy of identifying the current object. The following will refer to the appendix... Figures 7-8 Steps S610 to S630 will be described in more detail.

[0043] Figure 7 This is a schematic diagram illustrating the acquisition of the appearance and geometric features of an object according to an exemplary embodiment of the present invention. Figure 8 This is a flowchart illustrating the acquisition of the appearance and geometric features of an object according to an exemplary embodiment of the present invention.

[0044] Reference Figure 7 and Figure 8In step S810, based on the first point cloud features, the first appearance features and first geometric features of each object in the image can be obtained. This step can be performed by a first region candidate network. More specifically, as an example and not a limitation, the first point cloud features, which do not have clear physical meaning, are processed by the first region candidate network. Each candidate region corresponding to an object can obtain four different features, corresponding to the appearance (first appearance feature), center position (x, y, z), size (w (width), h (height), d (length)), and orientation (roll, pitch, yaw) of the candidate region object. The center position (x, y, z), size (w, h, d), and orientation (roll, pitch, yaw) of the object correspond to the first geometric features. Here, as an example and not a limitation, the first region candidate network can be a deep learning network based on convolutional neurons or a deep learning network with a multilayer perceptron (MLP) structure.

[0045] In step S820, the geometric weight between every two candidate regions can be obtained based on the first geometric features of each object in the image. More specifically, the geometric feature association vector between every two candidate regions can be obtained based on the first geometric features of each object in the image, and the geometric weight between every two candidate regions can be obtained based on the geometric feature association vector between every two candidate regions. Equation 1 below shows the geometric features (f) between two candidate regions m and n. G m f G n )Association vector ε G .

[0046] [Equation 1]

[0047]

[0048] dot(roll m , roll n ), dot(pitch) m pitch n ), dot(yaw m yaw n )]

[0049] Where x, y, z correspond to the center position features of the object output by the first region candidate network, w, h, d correspond to the size features of the object output by the first region candidate network, and roll, pitch, yaw correspond to the orientation features of the object output by the first region candidate network. x, y, z, w, h, d, roll, pitch, yaw with subscripts m and n represent the x, y, z, w, h, d, roll, pitch, yaw of candidate region m or n; εG It is a 6-dimensional vector that relates to the center position, size, and orientation of the object.

[0050] More specifically, the step of obtaining the geometric weights between candidate regions based on the geometric feature association vector between candidate regions may include: converting the 6-dimensional vector ε G Input into neural network W G To map to a new feature space, in order to compute two vectors f in the high-dimensional space (i.e., the aforementioned new feature space). G m f G n The distance is used as the geometric weight w between two regions m and n. G mn Here, the neural network W is used only as an example and not as a limitation. G It can be a deep learning network with a multilayer perceptron (MLP) structure. Furthermore, the obtained geometric weights w can be obtained through the following equation 2. G mn Take the positive value.

[0051] [Equation 2]

[0052]

[0053] In step S830, the appearance weight between every two candidate regions can be obtained based on the first appearance features of each object in the image. More specifically, the appearance features f of two candidate regions m and n can be used to... A m f A n Input into neural network W K W Q The regions are mapped to a new feature space, and then the appearance weight w between the two candidate regions m and n can be calculated using the following Equation 3. A mn Here, the neural network W is used only as an example and not as a limitation. K W Q It can be a deep learning network with a multilayer perceptron (MLP) structure.

[0054] [Equation 3]

[0055]

[0056] Where, d k It can be a predetermined constant, representing a predetermined multiple of the number of candidate regions.

[0057] In step S840, the geometric weights w can be used as a basis. G mn and appearance weight w Amn The relationship weight between any two candidate regions is obtained. More specifically, the relationship weight w between candidate regions can be obtained through the following Equation 4. mn .

[0058] [Equation 4]

[0059]

[0060] Where k can represent a natural number from 1 to K, and K can be the number of candidate regions.

[0061] In step S850, a second point cloud feature can be obtained based on the first point cloud feature and the relation weights, wherein the second point cloud feature may include association information. More specifically, the first point cloud feature f can be expressed as follows: Equation 5 m Input into neural network W V The second point cloud feature f is obtained by weighted summation. R (n), the second point cloud features have the same number and feature dimension as the original point cloud features. Here, it is only for example and not a limitation, the neural network W V It can be a deep learning network with a multilayer perceptron (MLP) structure.

[0062] [Equation 5]

[0063]

[0064] Where m can represent a natural number from 1 to K, and K can be the number of candidate regions.

[0065] In step S860, based on the second point cloud features, the second appearance features and second geometric features of each object in the image can be obtained, wherein this step can be performed by a second region candidate network. Here, as an example and not a limitation, the second region candidate network can be a deep learning network based on convolutional neurons or a deep learning network with a multilayer perceptron (MLP) structure, and the first region candidate network and the second region candidate network can use the exact same network structure and can share or not share network weights.

[0066] Figure 9 This is a block diagram illustrating an object detection apparatus according to an exemplary embodiment of the present invention.

[0067] like Figure 9 As shown, an object detection apparatus according to an exemplary embodiment of the present invention may include a feature acquisition unit 910 and an object determination unit 920.

[0068] The feature acquisition unit 910 can acquire the first point cloud features based on the point cloud data of the image.

[0069] The object determination unit 920 can determine at least one object in the image based on the first point cloud features.

[0070] Furthermore, the object detection apparatus according to an exemplary embodiment of the present invention may also include other units to respectively implement the functions corresponding to other steps of the object detection method according to an exemplary embodiment of the present invention, which will not be described in detail here.

[0071] Furthermore, according to exemplary embodiments of the present invention, an electronic device may also be provided, including a memory and a processor, wherein the memory stores computer-executable instructions, and when the instructions are executed by the processor, the aforementioned method is performed. According to exemplary embodiments of the present invention, a computer-readable medium may also be provided, having stored computer-executable instructions thereon, and when the instructions are executed, the aforementioned method is performed.

[0072] According to the present invention, in an electronic device, an object detection method can obtain output data for recognizing an image or point cloud features in an image by using image data as input data for an artificial intelligence model.

[0073] Artificial intelligence models can be obtained through training. Here, "obtained through training" means that a basic artificial intelligence model with multiple training data is trained using a training algorithm to obtain predefined operating rules or artificial intelligence models, which are configured to perform the required features (or objectives).

[0074] An artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and neural network computation is performed by calculating the results of the previous layer and the multiple weight values.

[0075] Visual understanding, like human vision, is a technology for recognizing and processing things, including, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, or image enhancement.

[0076] According to an exemplary embodiment of the present invention, point cloud completion can improve the quality of data used for object detection by improving the accuracy and robustness of object detection or recognition in cases of object occlusion, field of view truncation, and small objects by extracting and fusing information on the relationships between multiple objects. Based on the above solution, 6-DOF geometric estimation of 3D objects based on point cloud data can be performed to improve the efficiency, accuracy, and robustness of the system in augmented reality applications. The object detection method and apparatus can be used in augmented reality systems and depth images to detect and perform 6-DOF geometric estimation of objects of interest around the user. Based on the obtained detection results, the virtual objects rendered in the augmented reality system can be controlled to allow real and natural interaction between real and virtual objects.

[0077] Although the invention has been shown and described with reference to specific exemplary embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the claims and their equivalents.

Claims

1. An object detection method, comprising: Based on the point cloud data of the image, obtain the first point cloud features; Based on the first point cloud features, obtain the first appearance features and the first geometric features of each object in at least one object in the image; Based on the first appearance feature and first geometric feature of each object in the image, the relationship weight between every two candidate regions is obtained, where the candidate region corresponds to the object; Based on the first point cloud features and relation weights, the second appearance features and second geometric features of each object in the image are obtained; The at least one object in the image is determined based on the second appearance feature and second geometric feature of each object in the image. The process of obtaining the second appearance feature and second geometric feature of each object in the image includes: obtaining the geometric weight between every two candidate regions based on the first geometric feature of each object in the image; obtaining the appearance weight between every two candidate regions based on the first appearance feature of each object in the image; obtaining the relationship weight between every two candidate regions based on the geometric weight and the appearance weight; obtaining the second point cloud feature based on the first point cloud feature and the relationship weight; and obtaining the second appearance feature and second geometric feature of each object in the image based on the second point cloud feature.

2. The method as described in claim 1, wherein, Before acquiring the first point cloud features, the point cloud data of the image is completed by point cloud augmentation. Among them, obtaining the first point cloud feature based on the point cloud data of the image includes: obtaining the first point cloud feature based on the point cloud data after point cloud completion.

3. The method as described in claim 2, wherein, Point cloud completion of image point cloud data includes: Extracting high-dimensional features from point cloud data of an image; Point cloud data for point cloud completion is obtained based on high-dimensional features.

4. The method of claim 1, wherein, Obtaining the first point cloud features includes: Based on the point cloud data of the image, the first point cloud feature corresponding to the geometric center of the object is obtained.

5. The method of claim 4, wherein, The steps to obtain the geometric weights between every two candidate regions include: Based on the first geometric feature of each object in the image, obtain the geometric feature association vector between every two candidate regions; Based on the geometric feature association vector between every two candidate regions, the geometric weight between every two candidate regions is obtained.

6. The method of claim 5, wherein, The first geometric feature relates to at least one of the object's center position, size, and orientation; and / or The second geometric feature relates to at least one of the object's center position, size, and orientation; and / or The geometric feature association vector is related to at least one of the object's center position, size, and orientation.

7. The method of claim 1, further comprising: Based on the identified objects, virtual objects are rendered in the augmented reality system.

8. An electronic device comprising a memory and a processor, wherein the memory stores computer-executable instructions, which, when executed by the processor, perform the method of any one of claims 1 to 7.

9. A computer-readable medium having stored thereon computer-executable instructions, which, when executed, perform the method of any one of claims 1 to 7.

Images