Object detection system and method for augmented reality

Abstract

A method and system for object detection in an augmented reality environment is disclosed. An image of the environment is received and a point cloud is generated using the image. Depth values ​​are extracted using the point cloud. One or more candidate recognition profiles are received from a neural network. Each candidate recognition profile includes a first bounding box surrounding a first portion of the image, the first portion including at least a portion of a specific object, a label, and a recognition probability. A virtual geometry is generated by modifying scale information associated with the object recognition profile using the depth value, and a scale match probability is generated using the virtual geometry. The scale match probability is updated with the neural network. A final recognition profile is generated using the neural network. The final recognition profile includes a second bounding box and an updated label, the second bounding box surrounding a second portion of the image including the specific object.

Description

Technical Field

[0001] The present disclosure relates generally to augmented reality and, more particularly, but not by way of limitation, to object detection by neural networks augmented with augmented reality sensor data. Background Art

[0002] Augmented reality (AR) systems overlay virtual content on the user's view of the real-world environment. With the development of AR software development kits (SDKs), the mobile industry has brought smartphone AR into the mainstream. AR SDKs typically provide six degrees-of-freedom (6DoF) tracking capabilities. Users can scan the environment using the smartphone's camera, which performs visual inertial odometry (VIO) in real time. After continuously tracking the camera pose, virtual objects can be placed into the AR scene to create an illusion that real and virtual objects are fused together.

[0003] Some augmented reality applications aim to generate virtual objects that appear naturally in real-world environments. Generating virtual objects out of context (e.g., a bed in a bathroom) prevents augmented reality applications from generating naturally appearing objects. Therefore, it is important to recognize objects in real-world environments in such systems to improve augmented reality displays. Summary of the invention

[0004] Aspects of the present disclosure include a method for object detection in an augmented reality environment. The method includes: receiving an image of an environment by a camera of a mobile device; generating a point cloud by the mobile device, the point cloud indicating the distance of a specific object displayed in the image; extracting a depth value from the point cloud, the depth value indicating the distance between the camera and the specific object; receiving one or more candidate identification profiles from a trained neural network, each candidate identification profile including a first bounding box and an identification probability, the first bounding box surrounding a first portion of the image, the first portion including at least a portion of the specific object, the identification probability indicating the probability that the specific object corresponds to a specific label; receiving scale information corresponding to the specific label, the scale information indicating the geometry of any object corresponding to the specific label; modifying the scale information by using the depth value to generate a virtual geometry; generating a scale matching probability by comparing the first bounding box and the virtual geometry; updating the neural network using the scale matching probability; generating a final identification profile using the updated neural network, the final identification profile including a second bounding box and an updated label, the second bounding box surrounding a second portion of the image including the specific object, the updated label corresponding to the specific object.

[0005] Another aspect of the present disclosure includes a system comprising one or more processors and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the above method.

[0006] Another aspect of the present disclosure includes a non-transitory computer-readable medium including instructions, which, when executed by one or more processors, cause the one or more processors to perform the above method.

[0007] The present disclosure achieves a number of advantages over conventional techniques. For example, the provided methods and systems use a neural network that generates improved object detection and recognition, characterized by higher accuracy than conventional techniques. Through the operation of the neural network, scale matching using an augmented reality sensor can be performed to modify candidate regions identified by the neural network. This allows the neural network to generate improved object detection and recognition with higher accuracy. In addition, applying scale matching at various stages of the neural network can exclude some candidate objects from being considered by the neural network, which can reduce the processing resources consumed by the neural network and increase the processing speed of successive layers of the neural network.

[0008] Other areas of applicability of the present disclosure will become apparent from the detailed description provided below.It should be understood that the detailed description and specific examples, although indicating various embodiments, are intended only for illustrative purposes and are not intended to necessarily limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An example of a computer system including a depth sensor and a red, green, and blue (RGB) optical sensor for AR applications according to an embodiment of the present invention is shown.

[0010] Figure 2 An example block diagram of a neural network augmented with augmented reality sensor data according to an embodiment of the present invention is shown.

[0011] Figure 3 The use of the embodiment according to the present invention is shown Figure 2 An example process of performing object detection using an enhanced neural network of an embodiment.

[0012] Figure 4 An example block diagram of a neural network augmented with augmented reality sensor data according to an alternative embodiment of the present invention is shown.

[0013] Figure 5 The use of the embodiment according to the present invention is shown Figure 4 An example process of performing object detection using an enhanced neural network of an embodiment.

[0014] Figure 6 An example block diagram of a neural network augmented with augmented reality sensor data is shown according to certain embodiments of the present invention.

[0015] Figure 7 The use of the embodiment according to the present invention is shown Figure 6 An example process of performing object detection using an enhanced neural network of an embodiment.

[0016] Figure 8 An example block diagram of the execution layer of the scale matching process according to an embodiment of the present invention is shown.

[0017] Fig. 9 An example computer system according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0018] Various embodiments are described below. For the purpose of explanation, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may also be implemented without the specific details. In addition, well-known features may be omitted or simplified in order not to obscure the described embodiments.

[0019] Embodiments of the present disclosure relate to object detection in a neural network enhanced with augmented reality sensor data. For example, an image is received from a camera of a mobile device. Using the image, the mobile device can generate a pose (e.g., position and orientation) of the camera and a point cloud of a specific object displayed in the image. The mobile device can generate a depth value using the pose and point cloud of the camera, the depth value indicating the distance between the camera and the specific object. The mobile device can receive candidate recognition profiles for various parts of the image from the neural network. Each profile can include a first bounding box, a label, and a probability, the first bounding box surrounding a first part of the image including a part of the specific object, the label corresponding to the specific object, and the probability being the probability that the label corresponds to the specific object. The mobile device can receive scale information indicating the geometry of any object corresponding to the above-mentioned label. By modifying the scale information using the depth value, a virtual geometry can be generated. The mobile device can generate a scale matching probability using the virtual geometry, and the scale matching probability can update the neural network. The updated neural network outputs a final recognition profile, which includes a second bounding box and a new label, the second bounding box surrounding a second part of the image including the specific object, and the new label corresponding to the specific object.

[0020] For example, an AR application can use object detection to select the placement of a specific virtual object and / or a specific virtual environment in an AR scene. The AR application selects a specific virtual object or its placement based on the background of the real world environment so that the virtual object appears to belong to the real world environment. Object detection can be performed by an AR application, a neural network, or a neural network enhanced by AR sensor data. For example, a camera of a mobile device can capture an image of an office environment. The mobile device generates a point cloud of objects in the image, such as a table, a chair, etc., and calculates a depth value corresponding to at least one object (e.g., the distance between the camera and the chair). The neural network generates candidate recognition profiles for objects in the image. Each candidate recognition profile includes a bounding box surrounding a portion of the image, which corresponds to an object such as a chair. The candidate recognition profile can also include the probability that the portion of the image corresponds to a specific label (e.g., a chair). The mobile device can receive scale information for any chair. The scale information includes the average size of any chair (e.g., 0.5 to 1 meter high, 0.5 to 1 meter wide, etc.). The mobile device then generates virtual geometry by modifying the scale information using the depth value (e.g., taking into account the distance of the object from the camera).

[0021] The mobile device can then generate a scale match probability by comparing the bounding box of the candidate recognition profile to the virtual geometry. The scale match probability can indicate the accuracy of the recognition profile. For example, if the bounding box is close to the virtual geometry, the bounding box may be accurate. If the bounding box deviates from the virtual geometry, the recognition profile may be inaccurate.

[0022] The scale match probability can be used to update the neural network. For example, the probability of a recognition profile can be updated (e.g., the probability of corresponding to a particular label becomes higher or lower), or the probability of a recognition profile can be eliminated (e.g., when the scale match probability indicates that the recognition profile is very inaccurate), etc. The updated neural network provides a final recognition profile, which includes a second bounding box and a new label around the second part of the image (including the chair). In some cases, the second part of the image can be equal to the first part of the image. In other cases, the second part of the image can more accurately include the detected object (e.g., a chair). Similarly, the new label may be the same as the old label (if the candidate recognition profile includes the correct label), or the updated neural network may generate a new label that more accurately corresponds to the object (e.g., an office chair).

[0023] Scale matching (e.g., generation or application of scale matching probabilities) can be performed at various stages of the neural network for object detection. For example, scale matching can be performed after the last layer (e.g., after the non-maximum suppression process), before the last layer (e.g., before the non-maximum suppression process), and / or earlier in the neural network processing (e.g., after candidate region generation, after scale classifier generation, etc.).

[0024] The methods and systems described herein improve the accuracy of object detection in a neural network. The neural network uses scale matching from augmented reality sensor data to modify candidate objects in an image that the neural network is trying to identify. This allows the neural network to generate improved object detection and recognition with higher accuracy. Additionally, at various stages of the neural network, scale matching can exclude some candidate objects from the neural network, which can reduce the processing resources consumed by the neural network and increase the processing speed of successive layers of the neural network.

[0025] Figure 1An example of a computer system 110 according to an embodiment of the present invention is shown, the computer system 110 includes a depth sensor 112 and an RGB optical sensor 114 for an AR application. The AR application can be implemented by an AR module 116 of the computer system 110. Generally, the RGB optical sensor 114 generates an RGB image of a real-world environment, which includes, for example, a real-world object 130. The depth sensor 112 generates depth data about the real-world environment, wherein the data includes, for example, a depth map, which shows the depth of the real-world object 130 (e.g., the distance between the depth sensor 112 and the real-world object 130). After the AR session is initialized (which may include calibration and tracking), the AR module 116 renders an AR scene 120 of the real-world environment in the AR session, wherein the AR scene 120 can be presented in a graphical user interface (GUI) on the display of the computer system 110. The AR scene 120 displays a real-world object representation 122 of the real-world object 130. In addition, the AR scene 120 displays a virtual object 124 that does not exist in the real-world environment. The AR module 116 can generate a RGB and depth (RGBD) image from the RGB image and the depth map, which can be processed by the AR system to enhance the object detection neural network. For example, the AR application can use the posture of the computer system 110 and the point cloud of the object to obtain scale information of the candidate object category identified by the neural network. The scale information is passed to the neural network to correct the object detection, which improves the accuracy of object detection (and recognition) and reduces the processing resources consumed by the neural network.

[0026] In the example, the computer system 110 represents a suitable user device, and in addition to the depth sensor 112 and the RGB optical sensor 114, the computer system 110 also includes one or more graphics processing units (graphical processing units, GPUs), one or more general purpose processors (general purpose processors, GPPs), and one or more memories storing computer-readable instructions, which can be executed by at least one processor to perform various functions of the embodiments of the present disclosure. For example, the computer system 110 can be any one of a smartphone, a tablet computer, an AR headset, or a wearable AR device.

[0027] The AR module 116 can perform a visual-inertial odometry (VIO) process to track the pose (e.g., position and orientation) of the AR module 116. VIO uses image analysis and an inertial measurement unit (sensor data) to determine changes in the position and / or orientation of a camera (e.g., AR module 116). Visual odometers can use feature detection in images to identify and associate features in consecutive images. Feature detection can be used to generate an optical flow field that estimates the motion of a camera relative to an object displayed in consecutive images. The degree of motion between time intervals (e.g., the time interval between consecutive images) can be used to determine the distance and direction that the camera moved during the time interval. The distance and direction of the camera (and sensor data) can be used to track the position and orientation of the camera at each time interval. The visual odometer can be enhanced using an inertial measurement unit that captures directional force values.

[0028] For example, the AR module 116 can perform an implementation of VIO called a simultaneous localizations and mapping (SLAM) process. For example, the SLAM process can start with a calibration step, in which an empty map of the environment can be initialized so that the device is located at the origin of the coordinate system. The SLAM process receives input data, such as but not limited to image data, control data ct, sensor data st, and time interval t. The SLAM process then generates an output, which may include an approximate position xt of the device within a given time interval (related to one or more approximate positions of one or more previous time intervals) and a map mt of the environment. The output can be enhanced (or verified) using feature detection images captured at time t and time t+1 to identify and associate features in the image. Changes between images can be used to verify the movement of the AR module 116, fill the environment mt with objects detected in the image, and the like.

[0029] The SLAM process can update xt and mt when the device captures sensor data (and image data from the device's camera) indicating movement in a specific direction. The SLAM process can be an iterative process that updates xt and mt at set time intervals or when new sensor data or image data can be detected. For example, if no sensor changes occur between time intervals t and t+1, the SLAM process can delay updating the position and map to preserve processing resources. When the detected sensor data changes indicate that the device is likely to have moved from its previous position xt, the SLAM process can calculate the device's new position xt and update the map mt.

[0030] The AR module 116 may be implemented as dedicated hardware and / or a combination of hardware and software (e.g., a general purpose processor and computer readable instructions stored in a memory and executable by the general purpose processor). In addition to initiating an AR session and performing VIO, the AR module 116 may also use augmented reality sensor data to improve object detection of real-world objects. The AR module may determine the distance between the AR module 116 and the real-world object 130. During object detection by the neural network, the AR module 116 may update the neural network using candidate categories of the object (e.g., one or more categories that the neural network has determined to have the possibility of corresponding to the real-world object 130). For example, the AR module 116 may obtain the size of the object corresponding to the candidate label. The size is then modified according to the distance between the object and the camera determined by the AR module 116 (e.g., if the object is far away, the size will become smaller, and vice versa). Then, if the candidate bounding box of the label generated by the neural network is too large or too small for a given scale match, the probability that the bounding box of the label is correct may be reduced (or the bounding box and label may be eliminated).

[0031] exist Figure 1 In an illustrative example of , a smartphone is used to display an AR session of a real-world environment. In particular, the AR session includes rendering an AR scene that includes a representation of a real-world table with a vase (or some other real-world object) placed on the table. The augmented reality session can include rendering virtual objects, such as virtual object 124, superimposed on the AR scene. The AR session can include a neural network that recognizes objects in the real-world environment (e.g., to select virtual object 124, locate virtual object 124, etc.). AR module 116 initializes the AR session by using the image to determine an initial pose of the smartphone (e.g., determined using a SLAM process, etc.) and initializing a coordinate system relative to the smartphone.

[0032] The AR module 116 can generate a point cloud for an object (e.g., a real-world object 130) displayed in the image. The point cloud includes two or more points (e.g., coordinates in a coordinate system), each of which corresponds to a discrete point of the object. The point cloud represents the depth of the object (e.g., the distance of the object relative to the smartphone). The AR module 116 uses the point cloud and the posture of the smartphone to extract the depth value of the object.

[0033] The smartphone executes a neural network for object detection. The neural network uses the images captured by the camera to detect and identify the objects shown in the images. The neural network can generate one or more candidate recognition profiles. Each candidate recognition profile includes a bounding box and a probability, where the bounding box surrounds a portion of the image, the portion of the image includes a portion of the object, and the probability is the probability that the object corresponds to a specific label of the object. The bounding box can be represented by a six-dimensional (6D) vector, such as (c, x, y, z, w, g, p), where x, y, z are dimensions, w is width, h is height, c is the label, and p is the probability that the label is correct. For each candidate recognition profile, the AR module 116 uses the label to obtain scale information. The scale information includes the size of any object that conforms to the label. For example, if the label is a table, the scale information includes the average size of any table. Alternatively, the scale information may include the range of the table (minimum-maximum value in each dimension). The AR module 116 generates a virtual geometry for any object (e.g., a table). The virtual geometry is generated by modifying the scale information using the depth value.

[0034] The AR module 116 uses the comparison between the bounding box and the virtual geometry to generate a scale match probability. The scale match probability can be used to update the neural network by adjusting the probabilities assigned to candidate recognition profiles. For example, if the bounding box of the label "table" is larger than the virtual geometry, the predicted label of the candidate recognition profile may be incorrect because the predicted label should appear smaller at the distance measured by the AR module 116. Updating the probabilities can include removing candidate recognition profiles that deviate from the bounding box by more than a threshold amount (e.g., 10%, 15%, or any predetermined amount).

[0035] The updated neural network generates a final recognition profile for the real-world object 130. The final recognition profile may include an updated bounding box around the portion of the image that includes the table and an updated label (e.g., if different from the candidate recognition profile). The neural network may present the final recognition profile via a display of the smartphone. For example, the neural network (or AR module 116) may generate bounding boxes 126 and 128 with labels that indicate labels corresponding to corresponding bounding boxes within the displayed AR scene 120.

[0036] Figure 2An example block diagram of a neural network enhanced with augmented reality sensor data according to an embodiment of the present invention is shown. Object detection can use a combination of AR framework 204 and object detection neural network 220. For example, a computing device can execute AR framework 204 and object detection neural network 220 within an AR session. The computing device can be a mobile device (e.g., a smartphone, laptop, tablet, etc.), a desktop computer, a server, a thin client, etc. Object detection can utilize AR sensor data during the operation of object detection neural network 220 to improve the performance (e.g., accuracy, resource consumption, speed, etc.) of object detection neural network 220.

[0037] The AR framework 204 can operate according to the AR session. An input image can be received and used to continuously track the pose of the computing device 208 (e.g., using SLAM or other tracking). The AR framework can also calculate a point cloud 212 associated with surfaces and / or objects within the image (e.g., by using depth information included in the RGBD image, a separate depth sensor, or using stereo disparity of two images). The point cloud indicates the depth of each surface (e.g., the distance between the surface and the camera that acquired the image).

[0038] The object detection neural network 220 uses the same image received by the AR framework 204 to initiate the object detection process. In some cases, the object detection neural network 220 is initiated in parallel with the AR framework 204 determining the pose 208 and generating the point cloud 212. In other cases, the object detection neural network 220 is initiated before or after the AR framework determines the pose 208. The object detection neural network 220 generates candidate recognition profiles, which can be refined using the regressor 228. The regressor 228 refines the candidate region (e.g., the bounding box of the candidate recognition profile) for classification by the classifier. The classifier 224 generates a probability that the bounding box of the candidate recognition profile corresponds to a particular label. The object detection neural network 220 can perform a non-maximum suppression process to eliminate overlap (e.g., redundant candidate recognition profiles).

[0039] The remaining labels for each candidate recognition profile are passed to the AR framework. The AR framework uses the label to query the scale information database 216, which returns the scale information corresponding to the label. The scale information includes the dimensions (e.g., height, width, depth) of any object assigned the label. The scale information may include the minimum and maximum values ​​for each dimension. Example scale information is shown in Table 1 below. Alternatively, the scale information may include the object size mean and standard deviation for each size.

[0040] Label minL(m) minW(m) minH(m) maxL(m) maxW(m) maxH(m) airplane 3 2.18 1.5 84.0 117.0 24.2 car 2.4 1.2 1.2 5.6 2.0 2.0 Chair 0.4 0.45 0.8 0.85 0.9 1.4 mouse 0.13 0.19 0.05 0.08 0.05 0.03

[0041] Table 1 Examples of label scale databases

[0042] The AR framework 204 can process the scale information to generate a virtual geometry for the candidate recognition profiles. The virtual geometry can be generated by modifying the scale information using the measured depth values ​​of the pixels included in the bounding box (e.g., using the posture 208 and point cloud 212 of the computing device). The virtual geometry can be passed back to the object detection neural network 220, which can perform scale matching 232. Scale matching removes candidate recognition profiles whose bounding boxes differ from the virtual geometry by more than a threshold. Alternatively or additionally, scale matching 232 modifies the probability of each candidate recognition profile based on a comparison of the virtual geometry with the bounding box (e.g., increasing the probability when the bounding box is close to the virtual geometry and reducing the probability when the bounding box is greatly different from the virtual geometry). The neural network then selects the candidate recognition profile with the highest probability from the candidate recognition profiles as the output of the object detection neural network 220.

[0043] Figure 3 The use of the embodiment according to the present invention is shown Figure 2 An example process of object detection using a boosted neural network.

[0044] At block 304, one or more images are received. The images may include color data (eg, RGB values) and depth values. For example, the images may be received from an RGBD optical sensor or the like.

[0045] At block 308, the AR framework (eg, Figure 2 The AR framework 204 uses the one or more images to determine a pose of the computing device (e.g., determined using a SLAM process, etc.) and a point cloud of at least one surface displayed in the image. For example, the image may include depth data (e.g., from an RGBG optical sensor) indicating the distance between the point shown in the image and the camera. The AR framework uses the depth data to generate a plurality of points, each point corresponding to a discrete coordinate of a surface displayed in the image. The points may be represented by (three-dimensional) coordinates within a coordinate system of the computing device.

[0046] For example, an RGBD optical sensor can generate multi-dimensional (e.g., two-dimensional) data, where each pixel includes color data (e.g., RGB values) and a depth value. The AR framework can use the depth value of the pixel to convert the two-dimensional coordinates of each pixel into three-dimensional coordinates. These three-dimensional coordinates can form points in a point cloud (e.g., the number of points is equal to the resolution of the two-dimensional image). In some examples, the number of points can be less than the resolution of the image. In other examples, the number of points can be greater than the resolution of the image (e.g., when a point cloud is generated using multiple images using depth values ​​for each image and / or stereo disparity between images).

[0047] In box 312, the AR framework can extract a depth value for at least one surface displayed in the image. The AR framework can derive a single depth value from the points in the point cloud corresponding to the at least one surface (and optionally the posture). In some examples, because the point cloud may include multiple points for each surface, the AR framework can average the depth values ​​of the points of the surface. In other examples, the AR framework can select a minimum depth value. In other examples, the AR framework can select a maximum depth value. The AR framework can select any depth value based on predefined criteria (e.g., user-derived criteria).

[0048] At block 324 (in some embodiments substantially in parallel with block 308), a neural network (e.g. Figure 2 The object detection neural network 220 may begin processing one or more images to perform object detection. The object detection neural network may be a deep neural network, such as, but not limited to, a convolutional neural network (CNN), a single-shot multibox detector (SSD), you only look once (YOLO), a faster region-based CNN (faster R-CNN), a region-based fully convolutional network (R-FCN), etc. The neural network performs object detection in various layers, wherein the processing of a given layer may both indicate and be used by subsequent layers. The initial layer may identify candidate regions (e.g., potential regions of interest) within the image that may correspond to objects to be detected. The candidate regions may include bounding boxes surrounding the candidate regions. Subsequent layers may add, remove, or modify candidate regions and determine the probability that the candidate regions correspond to specific labels (e.g., generating candidate recognition profiles for the candidate regions). The last layer may include a non-maximum suppression process that eliminates candidate recognition profiles for a given candidate region that are less than a threshold amount.

[0049] For example, a neural network performs operations in one or more consecutive layers, such as, but not limited to, candidate region selection for identifying candidate regions of an image, object detection for assigning a label (e.g., a category) to each candidate region, a regressor (e.g., linear, polynomial, logarithmic, etc.) that refines two or more candidate regions (e.g., by performing a regression analysis) into a single candidate region, and a classifier (e.g., k-nearest neighbors (k-NN), softmax, etc.) that generates a candidate recognition profile that includes a bounding box (e.g., an identity of the candidate region) and a probability that the label (e.g., a label) corresponds to the image portion of the candidate region.

[0050] At block 328, the neural network may perform non-maximum suppression, which may remove some candidate recognition profiles. Non-maximum suppression organizes the candidate recognition profiles according to probability (e.g., from high to low or from low to high). The neural network then selects the candidate recognition profile with the highest probability. The neural network suppresses (e.g., removes) all candidate recognition profiles whose bounding boxes overlap a large portion of the bounding box of the selected candidate recognition profile (e.g., based on a predefined overlap level threshold). Alternatively or additionally, non-maximum suppression may suppress candidate recognition profiles whose probabilities do not exceed a threshold. The remaining candidate recognition profiles (or in some embodiments, only the labels) are used for scale matching.

[0051] In box 316, the AR framework uses the label of the candidate recognition profile to obtain scale information from the database and supplements the scale information with the depth value. For example, the AR framework can use the label to send a query to the database. The scale information database stores the scale information for each label. For each dimension (e.g., x, y, z), the scale information may include: 1) minimum and maximum values, or 2) average and standard deviation. The scale information database may return the scale information to the AR framework (or neural network). The AR framework may generate virtual geometry for each candidate recognition profile. The virtual geometry is generated by modifying the scale information using the depth value of the surface included in the bounding box of the candidate recognition profile (e.g., the depth value extracted in box 312). In other words, the AR framework receives the general size of any object assigned the label of the candidate recognition profile and modifies these sizes to take into account the distance of the object measured by the (AR framework) to the computing device. For example, if the depth value is high, the virtual geometry will be smaller than the scale information (because the object appears smaller to the user), and if the depth value is low, the virtual geometry will be larger than the scale information (because the object appears larger to the user).

[0052] The virtual geometry may be projected onto a two-dimensional camera view presented by a display of the computing device. In some cases, the virtual arbitrary object having the modified size is presented on the display. In other cases, a bounding box having a size corresponding to the modified size is displayed on the display.

[0053] At block 320, a scale matching process may be performed. The scale matching process compares the virtual geometry to the bounding box of each candidate recognition profile. A scale matching probability may be generated based on the comparison between the bounding box and the virtual geometry. Specifically, the more the bounding box covers the virtual geometry, the more accurate the bounding box may be. The value of the scale matching probability depends on the degree to which the bounding box covers the virtual geometry. In some examples, a high degree of coverage increases the value of the scale matching probability. A low degree of coverage decreases the value of the scale matching probability. In other examples, a high degree of coverage decreases the value of the scale matching probability. A low degree of coverage increases the value of the scale matching probability.

[0054] In some examples, the AR framework processes the scale information to generate virtual geometry and scale matching probabilities. In other examples, the neural network receives the scale information, the posture of the computing device, and the point cloud. The neural network then generates the virtual geometry and scale matching probabilities. In other examples, a combination of the AR framework and the neural network can be used to generate the virtual geometry and scale matching probabilities.

[0055] At block 332, for each remaining candidate recognition profile, the neural network determines an updated probability that the portion of the image within the bounding box corresponds to a particular label. The neural network may receive the candidate recognition profiles output from the non-maximum suppression process and the corresponding scale match probability for each candidate recognition profile. The scale match probability may be used to update the probability of the corresponding candidate recognition profile. For example, if the scale match probability is low (e.g., there is little overlap between the virtual geometry and the bounding box), the probability of the candidate recognition profile may be reduced, so that the candidate recognition profiles may be effectively eliminated and a candidate recognition profile whose bounding box better overlaps with the scale match probability may be selected.

[0056] For example, a first candidate recognition profile may include a bounding box and a label corresponding to a table. The depth value of the image portion indicates that the object is farther away, so the virtual geometry of the object is smaller than the bounding box. The probability of the candidate recognition profile will be reduced to reflect that the size of the candidate object is inconsistent with the measurement of the computer device.

[0057] The neural network selects the candidate recognition profile with the highest (modified) probability from the remaining candidate recognition profiles. The neural network then outputs a bounding box, a label, and (optionally) the probability that the portion of the image surrounded by the bounding box corresponds to the label.

[0058] At block 336, the output bounding box is displayed by overlaying the bounding box on the view of the AR scene. The bounding box can be presented with a label and a probability that the portion of the image bounded by the bounding box corresponds to the label. In some cases, the AR framework tracks the portion of the image bounded by the bounding box so that the bounding box continues to be displayed on the correct portion of the image even if the view of the environment changes (e.g., the camera moves).

[0059] Figure 4 An example block diagram of a neural network augmented with augmented reality sensor data according to an alternative embodiment of the present invention is shown. Where appropriate, Figure 2 The discussion of neural networks augmented with augmented reality sensor data provided in Figure 4 The neural network can be updated with AR sensor data at various stages of the neural network. Figure 2 The block diagram (and Figure 3 ) shows the modifications that occur after the non-maximum suppression process (e.g., in the last layer of the neural network before generating an output). Updating the neural network at an earlier stage can reduce the number of candidate regions processed by the neural network (e.g., eliminating regions and / or labels with low probability of being accurate), thereby improving the resource consumption of the neural network and increasing the speed at which the neural network processes images. For example, the object detection neural network 420 combines the scale matching 408 from the AR sensor data after the object detection layer 404 and before the non-maximum suppression 412 (e.g., as described above in conjunction with Figure 2 The AR framework 204, pose 208, point cloud 212 and scale information database 216).

[0060] The object detection layer 404 of the neural network generates candidate recognition profiles for all possible candidate regions in the image without suppressing redundant (e.g., overlapping) profiles with non-maximum probabilities within local regions. Scale matching 408 can be performed after the operation of the detection layer to filter out candidate recognition profiles with a low probability that the portion of the image surrounded by the boundaries of the candidate recognition profiles corresponds to the assigned label. Because these candidate recognition profiles are deleted before non-maximum suppression, non-maximum suppression can process fewer candidate recognition profiles, thereby increasing the speed at which non-maximum suppression is performed and the speed at which output can be obtained. In addition, reducing candidate recognition profiles early in the process can prevent inaccurate candidate recognition profiles from biasing the remaining layers of the neural network (e.g., reducing the accuracy of the neural network).

[0061] Figure 5 The use of the embodiment according to the present invention is shown Figure 4 Example process of object detection using an enhanced neural network. Blocks 304-320 are similar to combining Figure 3 The box described and, where applicable, the Figure 3The description also applies to Figure 5 .

[0062] At block 504, the neural network processes the one or more images using various layers of the neural network. The neural network performs an object detection layer (e.g., Figure 4 The object detection layer 404 generates one or more candidate recognition profiles (e.g., each candidate recognition profile includes a bounding box around a candidate region of the image and a probability that the candidate region corresponds to a particular label). The labels and probabilities may be output from a trained classifier layer that identifies features in the candidate region that are similar or identical to features in the training dataset. The output of the object detection layer is passed to classification at block 508 and scale information retrieval at block 316. Figure 5 In the example process of FIG. 3 , scale information retrieval at block 316 is initiated using all possible candidate regions of the image without suppressing redundant (eg, overlapping) regions with non-maximum probability within the local region.

[0063] At block 508, the neural network determines, for each remaining candidate recognition profile, an updated probability that the portion of the image within the bounding box corresponds to a particular label. The scale match probabilities retrieved from block 320 may be used to update the probabilities of the corresponding candidate recognition profiles. Since the neural network has not yet applied the non-maximum suppression process, the probabilities (and labels) of all possible candidate regions of the image (e.g., including redundant regions) are updated using the corresponding scale match information from block 320. In some examples, candidate recognition profiles with probabilities below a threshold may not be further processed.

[0064] At block 512, the neural network may apply non-maximum suppression to the candidate recognition profiles. Non-maximum suppression organizes the candidate recognition profiles according to probability (e.g., from high to low or from low to high). The neural network then selects the candidate recognition profile with the highest probability. The neural network suppresses (e.g., removes) all candidate recognition profiles whose bounding boxes overlap a large portion of the bounding box of the selected candidate recognition profile (e.g., based on a predefined overlap level threshold). Alternatively or in addition, non-maximum suppression may suppress candidate recognition profiles whose probabilities do not exceed a threshold. By updating the probabilities of the candidate recognition profiles (e.g., at block 508), the candidate recognition profiles are compared with the candidate recognition profiles in the example of FIG. 514. Figure 3 The non-maximum suppression may suppress more candidate recognition profiles than the non-maximum suppression performed by block 328 of FIG. 1. This improves the accuracy of the neural network because fewer candidate recognition profiles remain after the non-maximum suppression process (e.g., there may be only one). If more than one candidate recognition profile remains, the neural network selects the candidate recognition profile with the highest probability from among the candidate recognition profiles.

[0065] At block 516, the bounding box, label, and probability (optional) of the selected candidate recognition profile are displayed by overlaying the bounding box on the view of the AR scene. In some cases, the AR framework tracks the portion of the image enclosed by the bounding box so that the bounding box continues to be displayed on the correct portion of the image even if the view of the environment changes (e.g., the camera moves).

[0066] Figure 6 An example of a block diagram of a neural network augmented with augmented reality sensor data according to certain embodiments of the present invention is shown. In some examples, the neural network can use two-stage object detection (e.g., Faster R-CNN). When appropriate, Figure 2 and Figure 4 The discussion of neural networks augmented with augmented reality sensor data in Figure 6 . The first stage identifies a large number of candidate regions, and the second stage performs object detection / classification on the candidate regions (e.g., performed using subsequent layers of the neural network). In this example, scale matching can be performed after the first stage, thereby reducing the number of candidate regions processed by subsequent neural network layers (e.g., reducing the processing load and resource consumption of subsequent neural network layers). In other examples, such as in a single-stage object detection network, a multi-scale feature map can be used to generate a classifier to detect objects at multiple scales (e.g., SSD). In this example, scale matching can be used to select a classifier with an appropriate scale for detection. For example, for each layer, if the bounding box appears too small or too large based on the measured real-world scale (e.g., determined by performing scale matching using point clouds, the posture of the computing device, and scale information), the detection classifier can be omitted for these layers.

[0067] In a two-stage object detection neural network, the object detection neural network 620 includes a region proposal layer 604 (e.g., stage one), which generates a large number of candidate regions for subsequent processing layers. Scale matching can be used to reduce the candidate regions. For example, for each candidate region, the scale information can be found using a label assigned to the candidate region. The scale information can be modified using depth values ​​(e.g., using the pose 208 and point cloud 212 in the AR framework 204) to generate a virtual geometry for the candidate region. If the bounding box of the candidate region is significantly larger or smaller than the virtual geometry (e.g., using a predetermined threshold), the neural network may not process the candidate region subsequently. Subsequent layers of the neural network may include object detection layers, classifiers, non-maximum suppression processes 612, and combinations of the above, etc., which can be used to output bounding boxes and labels for image regions.

[0068] In a single-stage object detection neural network, the object detection neural network 620 includes one or more layers (rather than an entire stage) for identifying the candidate regions 604. The object detection neural network 620 may include multiple scale classifiers 608 that may detect a particular feature in an image at one or more scales (e.g., each scale corresponding to the size of the feature at a different distance from a camera of a computing device). A multi-scale feature map (e.g., a map including multiple scales for each feature) may be used to generate (e.g., train) a classifier to detect features at a particular scale. In general, because the scale may be unknown, each classifier may be applied to detect a feature (e.g., each classifier may be applied via a different layer of the neural network).

[0069] Scale matching can be used to determine scales that are inconsistent with AR sensor data (e.g., bounding boxes are too small or too large, etc.). Classifiers corresponding to these scales can be omitted, which can reduce the number of layers of the neural network. Subsequent layers of the neural network can include object detection layers, classifiers, non-maximum suppression processes 612, and combinations thereof. Subsequent layers can be used to output bounding boxes and labels for image regions.

[0070] The AR framework 204 can operate in accordance with the AR session. An input image can be received and used to continuously track a pose 208 of a computing device (e.g., using SLAM, etc.). The AR framework can also compute a point cloud 212 associated with surfaces and / or objects within the image (e.g., by using depth information included in an RGBD image, a separate depth sensor, or using stereo disparity of two images). The point cloud indicates the depth of each surface (e.g., the distance of the surface from a camera that acquired the image).

[0071] The label of each candidate recognition profile can be output to the AR framework. The AR framework can use the label to send a query to the scale information database 216, which returns the scale information corresponding to the label. The scale information includes the size (e.g., height, width, and depth) of any object assigned the label.

[0072] The AR framework 204 can process the scale information to generate a virtual geometry of the candidate recognition profiles. The virtual geometry can be generated by modifying the scale information using measured depth values ​​of pixels included in the bounding box (e.g., using the pose 208 and point cloud 212 of the computing device). The virtual geometry can be passed back to the object detection neural network 220, which can perform scale matching 232. Scale matching 232 can be performed to reduce the number of candidate regions (e.g., candidate recognition profiles) to be processed by subsequent layers of the object detection neural network 620, or to eliminate scale classifiers corresponding to scales that are inconsistent with the AR sensor data.

[0073] Figure 7 The use of the embodiment according to the present invention is shown Figure 6 Example process of object detection using an enhanced neural network. Blocks 304-320 are similar to combining Figure 3 A box describing, where appropriate, Figure 3 The description provided also applies to Figure 7 . Figure 7 The description of the example process of begins at block 704 , where the neural network may begin processing the input image received from block 304 .

[0074] At block 708, depending on the type of neural network, candidate regions may be generated and / or a scale classifier may be generated. For example, a two-stage object detection neural network generates a large number of candidate regions (e.g., candidate recognition profiles). A single-stage object detection neural network may generate candidate regions and use a multi-scale feature map to generate a scale-based classifier.

[0075] At block 316, scale information is obtained from the database using the label assigned to the candidate recognition profile. The scale information may be supplemented by depth (e.g., from block 312) to generate a virtual geometry (e.g., the size of any object assigned the label, scaled to correspond to the distance of the portion of the image enclosed by the bounding box from the camera).

[0076] At block 320, scale matching may be performed, wherein a determination is made as to whether the bounding box of the candidate recognition profile is greater than, less than, or equal to the virtual geometry. For a two-stage object detection neural network, the correspondence between the virtual geometry and the bounding boxes of the candidate recognition profiles may be used to exclude some candidate recognition profiles from further processing by the neural network. For example, if the bounding box is greater than a first predetermined threshold or less than a second predetermined threshold, the candidate recognition profile may be deleted. For a single-stage object detection neural network, scale matching is used to identify scales that are inconsistent with the scale of the portion of the image enclosed by the bounding box. Processing of these candidate recognition profiles may be limited to scale classifiers (e.g., in subsequent layers) corresponding to scales similar to the portion of the image enclosed by the bounding box (size at a particular distance from the camera).

[0077] At block 712, scale matching may be used to determine an updated probability that the portion of the image enclosed by the bounding box corresponds to a particular (or possibly new) label. The scale matching information may be used to update the probability.

[0078] At block 716, the candidate recognition profiles are processed by subsequent layers of the neural network. The subsequent layers of the neural network may include object detection layers, classifiers, non-maximum suppression process 612, combinations thereof, and the like.

[0079] At block 720, the neural network may generate an output that includes a bounding box around a portion of the image and a label corresponding to the portion of the image. The output may also include a probability that the portion of the image corresponds to the label.

[0080] Figure 3 , Figure 5 and Figure 7 The example processes described in the foregoing are performed in conjunction with a computer system, which is an example of the computer system described above. Some or all of the blocks of the above-described processes may be implemented by specific hardware on the computer system and / or may be implemented as computer-readable instructions stored on a non-transitory computer-readable medium of the computer system. The stored computer-readable instructions represent programmable modules including codes that can be executed by a processor of the computer system. The execution of such instructions configures the computer system to perform the corresponding operations. Each programmable module in conjunction with the processor represents a device for executing the corresponding block. Although the blocks are shown in a particular order, it should be understood that a particular order is not required and one or more operations may be omitted, skipped, and / or reordered.

[0081] Figure 8 An example block diagram of the execution layers of a scale matching process according to an embodiment of the present invention is shown. Scale matching can operate using three layers, a user input layer (where the user interacts with the AR framework), a neural network layer, and an AR framework. The user input layer includes a scale database. Scale information for each tag can be added to the tag scale database 802 via user input. For example, when a new tag is defined, a user can generate a new entry in the database indicating the size of any object assigned to the tag. Alternatively or additionally, scale information can be automatically added from other network sources (e.g., the Internet, another database, etc.). The tag scale database (together with the tag 'c' 806 of the boundary polygon from the candidate recognition profile 810) is used to identify the geometry (e.g., scale information) of any object assigned to the tag 'c' 806.

[0082] The AR framework can use the geometric information to generate a virtual geometry including a bounding polygon 808, which includes a label 'c' 806, vertices (e.g., size), and a confidence value that the bounding polygon is accurate. The size of the bounding polygon 808 is determined based on the geometry 804, the size 824 of the candidate recognition profile, the camera pose 816, and the point cloud 812.

[0083] The bounding polygon 808 may be compared to the size 824 of the candidate recognition profile, which the neural network may use to perform a scale match probability 'S' 820. The scale match probability 'S' 820 may be used to generate (at a neural network layer) an updated probability P' 824 for the candidate recognition profile using the probability 'p' 828 and the scale match probability 'S' 820. In some cases, the updated probability P' may be calculated by P' = p*S. The updated probability may then be used in subsequent layers of the neural network to generate outputs.

[0084] Fig. 9 An example of components of a computer system 900 according to at least one embodiment of the present disclosure is shown. The computer system 900 is an example of the computer system described above. Although these components are shown as belonging to the same computer system 900, the computer system 900 may also be distributed.

[0085] The computer system 900 includes at least a processor 902, a memory 904, a storage device 906, an input / output (I / O) peripheral device 908, a communication peripheral device 910, and an interface bus 912. The interface bus 912 is used to communicate, send, and transmit data, control, and commands between the various components of the computer system 900. The memory 904 and the storage device 906 include computer-readable storage media, such as RAM, ROM, electrically erasable programmable read-only memory (EEPROM), hard disk drives, CD-ROMs, optical storage devices, magnetic storage devices, electronic non-volatile computer storage devices (such as flash memory), and other tangible storage media. Any such computer-readable storage medium can be used to store instructions or program codes for implementing various aspects of the present disclosure. The memory 904 and the storage device 906 also include computer-readable signal media. Computer-readable signal media include propagated data signals containing computer-readable program codes. Such propagated signals take any of a variety of forms, including but not limited to electromagnetic, optical, or any combination thereof. Computer readable signal media include any computer readable media that is not a computer readable storage media and that can communicate, propagate, or transport a program for use in conjunction with the computer system 900 .

[0086] In addition, the memory 904 includes an operating system, a program, and an application program. The processor 902 is used to execute stored instructions, and includes, for example, a logic processing unit, a microprocessor, a digital signal processor, and other processors. The memory 904 and / or the processor 902 can be virtualized and can be hosted in another computer system such as a cloud network or a data center. The I / O peripherals 908 include user interfaces, such as keyboards, screens (e.g., touch screens), microphones, speakers, other input / output devices, and computing components, such as graphics processing units, serial ports, parallel ports, universal serial buses, and other input / output peripherals. The I / O peripherals 908 are connected to the processor 902 by any port coupled to the interface bus 912. The communication peripherals 910 are used to facilitate communication between the computer system 900 and other computer systems through a communication network, and include, for example, a network interface controller, a modem, wireless and wired interface cards, antennas, and other communication peripherals.

[0087] Although the subject matter has been described in detail for its specific embodiments, it should be understood that those skilled in the art, after obtaining an understanding of the foregoing, can easily produce changes, variations and equivalents to these embodiments. Therefore, it should be understood that the present disclosure is presented for the purpose of illustration rather than limitation, and does not exclude the inclusion of modifications, variations and / or additions to the subject matter that are obvious to ordinary technicians. Skills in this field. In fact, the methods and systems described herein can be embodied in a variety of other forms. In addition, without departing from the spirit of the present disclosure, various omissions, substitutions and changes can be made to the form of the methods and systems described herein. The attached claims and their equivalents are intended to cover such forms or modifications that fall within the scope and spirit of the present disclosure.

[0088] Unless expressly stated otherwise, it should be understood that terms such as "process," "compute," "determine," and "identify" used in the discussion of this specification refer to the actions or processes of a computer system (e.g., one or more computers or similar electronic computer systems or devices) that manipulates or transforms data represented as physical electronic or magnetic quantities in the memory, registers, or other information storage devices, transmission devices, or display devices of a computing platform.

[0089] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computer system may include any suitable arrangement of components that provide a result conditioned on one or more inputs. Suitable computer systems include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computer system from a general-purpose computing device to a dedicated computing device that implements one or more embodiments of the present subject matter. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software used to program or configure a computer system.

[0090] Embodiments of the methods disclosed herein can be performed in the operation of such a computer system. The order of the blocks presented in the above examples can be changed, for example, the blocks can be reordered, combined and / or decomposed into sub-blocks. Some blocks or processes can be executed in parallel.

[0091] Conditional language used herein, such as "may," "could," "might," "for example," and the like, unless expressly stated otherwise or otherwise understood in the context of use, is generally intended to convey that some examples include and other examples do not include certain features, elements, and / or steps. Thus, such conditional language generally does not mean that one or more examples require features, elements, and / or steps in any way, or that one or more examples must include features, elements, and / or steps for determining whether such features, elements, and / or steps are included or will be performed in any particular example, with or without author input or prompting.

[0092] The terms "including", "having", etc. are synonymous and are used inclusively in an open manner and do not exclude other elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusiveness (not exclusiveness), for example, when used to connect a list of elements, the term "or" means one, some or all of the elements in the list. "For" or "configured to" used herein refers to open and inclusive language, and does not exclude devices used or configured to perform additional tasks or steps. In addition, the use of "based on" means openness and inclusiveness, because the process, step, calculation or other action "based on" one or more listed conditions or values ​​may actually be based on additional conditions or beyond the listed values. Similarly, the use of "based at least in part" means openness and inclusiveness, because the process, step, calculation or other action "based at least in part on" one or more listed conditions or values ​​can be based on additional conditions or values ​​outside the listed in practice. The titles, lists and numbers included here are only for ease of explanation and are not meant to be limiting.

[0093] The various features and processes described above can be used independently of each other, or can be used in combination in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular order, and the blocks or states associated therewith may be performed in other appropriate orders. For example, the blocks or states described may be performed in an order different from that specifically disclosed, or multiple blocks or states may be combined in a single block or state. Example blocks or states may be performed serially, in parallel, or in some other manner. Blocks or states may be added to or deleted from the disclosed examples. Similarly, the example systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed examples.

Claims

1. An object detection method for augmented reality, characterized in that: include: The camera of the mobile device receives an image of the environment; The mobile device generates a point cloud, the point cloud indicating a distance from a specific object displayed in the image to the camera; extracting a depth value from the point cloud, the depth value indicating the distance between the camera and the specific object; receiving one or more candidate recognition profiles from a trained neural network, each candidate recognition profile comprising a first bounding box surrounding a first portion of the image, the first portion of the image comprising at least a portion of the specific object, and a recognition probability indicating a probability that the specific object corresponds to a specific label; receiving scale information corresponding to the specific tag, the scale information indicating a geometry of any object corresponding to the specific tag; generating a virtual geometric figure by modifying the scale information using the depth value; generating a scale matching probability by comparing the first bounding box and the virtual geometric figure; updating the neural network using the scale matching probability; as well as A final recognition profile is generated using the updated neural network, the final recognition profile including a second bounding box surrounding a second portion of the image including the specific object and an updated label corresponding to the specific object.

2. The method according to claim 1, characterized in that The scale information includes a height value, a length value, and a width value of the arbitrary object corresponding to the specific tag.

3. The method according to claim 1, characterized in that The first bounding box is displayed by the mobile device.

4. The method according to claim 1, characterized in that: The second bounding box is displayed by the mobile device.

5. The method according to claim 3, characterized in that: The size of the second bounding box is based on the scale information.

6. The method according to claim 1, characterized in that The updating of the neural network using the scale matching probability comprises: updating the neural network by updating a recognition probability of at least one of the one or more recognition profiles, The method further comprises: determining that the updated recognition probability is below a threshold; and The trained neural network is executed to provide a new recognition profile.

7. The method according to claim 1, characterized in that The scale information is received before the trained neural network performs a non-maximum suppression process.

8. The method according to claim 1, characterized in that The trained neural network generates a plurality of recognition profiles, and the scale information is used to delete recognition profiles with probabilities lower than a threshold.

9. The method according to claim 1, characterized in that: The trained neural network generates a plurality of recognition profiles, and the scale information is used to combine two or more recognition profiles.

10. An object detection method for augmented reality, characterized in that: include: The camera of the mobile device receives an image of the environment; The mobile device generates a point cloud based on the image, the point cloud indicating a distance from at least one object in the image to the camera; Receiving one or more recognition profiles from a neural network, each recognition profile comprising a bounding box surrounding a portion of the image, a label corresponding to the portion of the image, and a probability that the portion of the image corresponds to the label; receiving scale information, the scale information including the size of any object corresponding to the tag; updating the neural network using the scale information and the point cloud; as well as A second bounding box and an updated label are generated using the updated neural network, the second bounding box surrounding a second portion of the image, the updated label corresponding to an object displayed in the second portion of the image.

11. The method according to claim 10, characterized in that Updating the neural network includes modifying a probability of at least one of the one or more recognition profiles.

12. The method according to claim 10, characterized in that Updating the neural network comprises: Based on the scale information and a bounding box of at least one recognition profile among the one or more recognition profiles, the at least one recognition profile is deleted.

13. The method according to claim 10, characterized in that Updating the neural network comprises: At least two identification profiles are merged.

14. The method according to claim 10, characterized in that Before the neural network performs a non-maximum suppression process, the neural network is updated.

15. An object detection system for augmented reality, characterized in that: include: one or more processors; as well as A non-transitory machine-readable medium comprising a program, which, when executed by the one or more processors, causes the one or more processors to perform the following operations: Receiving an image of the environment via a camera of the mobile device; generating, by the mobile device, a point cloud indicating a distance from a specific object displayed in the image to the camera; extracting a depth value from the point cloud, the depth value indicating the distance between the camera and the specific object; Receiving one or more candidate recognition profiles from a trained neural network, each candidate recognition profile comprising a first bounding box surrounding a first portion of the image, the first portion of the image comprising at least a portion of the specific object, and a recognition probability indicating a probability that the specific object corresponds to a specific label; receiving scale information corresponding to the specific tag, the scale information indicating a geometry of any object corresponding to the specific tag; generating a virtual geometric figure by modifying the scale information using the depth value; generating a scale matching probability by comparing the first bounding box and the virtual geometric figure; updating the neural network using the scale matching probability; as well as A final recognition profile is generated using the updated neural network, the final recognition profile including a second bounding box surrounding a second portion of the image including the specific object and an updated label corresponding to the specific object.

16. The system according to claim 15, characterized in that The scale information includes a height value, a length value, and a width value of the arbitrary object corresponding to the specific tag.

17. The system according to claim 15, characterized in that The first bounding box is displayed by the mobile device.

18. An object detection system for augmented reality, characterized in that: include: one or more processors; as well as A non-transitory machine-readable medium comprising a program, which, when executed by the one or more processors, causes the one or more processors to perform the following operations: Receiving an image of the environment via a camera of the mobile device; generating, by the mobile device, a point cloud based on the image, the point cloud indicating a distance from at least one object in the image to the camera; Receiving one or more recognition profiles from a neural network, each recognition profile comprising a bounding box surrounding a portion of the image, a label corresponding to the portion of the image, and a probability that the portion of the image corresponds to the label; receiving scale information, the scale information including the size of any object corresponding to the tag; updating the neural network using the scale information and the point cloud; as well as A second bounding box and an updated label are generated using the updated neural network, the second bounding box surrounding a second portion of the image, the updated label corresponding to an object displayed in the second portion of the image.

19. The system according to claim 18, characterized in that Updating the neural network includes modifying a probability of at least one of the one or more recognition profiles.

20. The system according to claim 18, characterized in that Updating the neural network comprises: Based on the scale information and a bounding box of at least one recognition profile among the one or more recognition profiles, the at least one recognition profile is deleted.

Images