Object detection methods, systems, and computer-readable media

Abstract

A method, system and storage medium for object detection are presented. The method includes; creating object representation data based on outputs from a neural network and augmented reality (AR); constructing a semantic map comprising object superpoints; determining a set of object superpoints in the semantic map whose locations are within a certain distance of a new incoming object point; updating the semantic map in response to the new incoming object point; and modifying a probability of a class to which the new incoming object point belongs as identified by the neural network based on the updated semantic map. The semantic map is composed of object superpoints having a list of scores corresponding to detected labels, a list of viewing directions, and a list of sizes. Based on the semantic map, the accuracy of object detection is improved by modifying the probability of an object label or class from the neural network.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an object detection method, system, and computer-readable medium. Background Technology

[0002] Object detection plays a crucial role in Augmented Reality (AR). The perception / understanding of objects in real-world scenes can bring numerous applications to AR. For example, the appearance of real objects can be altered by adjusting virtual overlays accordingly, and virtual objects can be placed into the scene using certain association rules (e.g., displaying a matching virtual chair near a real table). In e-commerce applications, relevant products can also be recommended based on an understanding of the scene.

[0003] Deep Neural Networks (DNNs), including SSD, YOLO, and Fast R-CNN, have been used for object detection in images and videos. Furthermore, with advancements in mobile integrated chips and dedicated DNN solutions on mobile devices, more and more DNNs can run in real-time on smartphones. However, DNNs for object detection require large amounts of training data to handle variations in size and viewpoint. In addition, traditional 2D object detection DNNs on mobile devices typically have relatively low mean average precision (mAP), for example, MobileNetV2_SSDLite achieves 22.1% mAP, indicating numerous false positives during the inference process. Using geometric information from CAD models or depth cameras can improve accuracy. However, obtaining CAD models is not easy for many real-world objects, and depth cameras on mobile devices are quite power-intensive, expensive, and have a limited working range (<5 meters).

[0004] On the other hand, mobile AR frameworks have become mainstream, such as Apple's ARKit and Google's ARCore. These AR frameworks employ SLAM algorithms (more specifically, VIO algorithms) to track camera pose (i.e., position and orientation) with six degrees of freedom (DoF). These frameworks also provide 3D sparse point cloud data, enabling the reconstruction of 3D data points at distances of more than 50 meters from the camera. Summary of the Invention

[0005] One object of this application is to provide an object detection method, system, and computer-readable medium for using semantic graphs to improve the accuracy of object detection.

[0006] In the first aspect of this application, an object detection method includes:

[0007] The processor creates object representation data based on the output from the neural network and augmented reality (AR), wherein the object representation data includes object label information of objects identified by the neural network in the image, the three-dimensional position of object points, and viewpoint and size information of the object points from the AR frame;

[0008] A semantic graph including object superpoints is constructed by the processor, wherein each object superpoint is represented by the historical score data of the object point, the viewpoint information and the size information;

[0009] The processor determines a set of object superpoints in the semantic graph, wherein the positions of the object superpoints in the set are within a specific distance of the newly entered object point, and the specific distance is determined based on the size of the object category identified by the neural network, to which the newly entered object point belongs;

[0010] The semantic graph is updated by responding to the newly arrived object point, wherein the scores of object superpoints in the groups determined in the semantic graph are updated based on the information of the newly arrived object point; and

[0011] The processor modifies the probability of the neural network identifying the category to which the newly arrived object point belongs based on the updated semantic map.

[0012] According to one embodiment of the first aspect of this application, the semantic map is constructed based on the object points, the projections of the object points onto the image being located within the boundary region of the object identified by the neural network.

[0013] According to one embodiment of the first aspect of this application, the object superpoint of the semantic graph is constructed using the projection position on the image at the midpoint of the object point within the boundary region of the object.

[0014] According to one embodiment of the first aspect of this application, the specific distance is the maximum size of the category to which the newly arrived object point belongs.

[0015] According to an embodiment of the first aspect of this application, updating the semantic graph in response to the newly arrived object point includes:

[0016] The score of the newly arrived object point is calculated by comparing its viewpoint information with the historical viewpoint information of the superpoints of the objects in the group and / or comparing its size information with the historical size information of the superpoints of the objects in the group. The newly arrived object point belongs to a category with a certain probability identified by the neural network, and the score of the newly arrived object point represents the change in the probability of the category to which the newly arrived object point belongs.

[0017] According to an embodiment of the first aspect of this application, when calculating the score of the newly arrived object point, the score of the newly arrived object point is calculated based on a first weight and a second weight, wherein the first weight is associated with the minimum angular difference between the viewpoint corresponding to the newly arrived object point and all viewpoints corresponding to the object superpoint in the group, and the second weight is associated with the minimum size difference between the size corresponding to the newly arrived object point and all sizes corresponding to the object superpoint in the group.

[0018] According to an embodiment of the first aspect of this application, if the minimum angle difference is less than a first predetermined angle, then the first weight is set to a first value;

[0019] If the minimum angle difference is greater than the second predetermined angle, then the first weight is set to the second value, where the first value is less than the second value and the first predetermined angle is less than the second predetermined angle.

[0020] According to an embodiment of the first aspect of this application, if the minimum size difference is within a predetermined range, the second weight is proportional to the minimum size difference; if the minimum size difference exceeds the predetermined range, the second weight is set to a fixed value.

[0021] According to one embodiment of the first aspect of this application, the score of the new object point increases as the minimum angle difference and / or the minimum size difference increases; the score of the new object point decreases as the minimum angle difference and / or the minimum size difference decreases, and an increase in the minimum angle difference and / or the minimum size difference indicates an increased chance of using the class probability of the new object point obtained from the neural network, and a decrease in the minimum angle difference and / or the minimum size difference indicates a decreased chance of using the class probability of the new object point obtained from the neural network.

[0022] According to an embodiment of the first aspect of this application, updating the semantic graph in response to the newly arrived object point includes:

[0023] For the category of the superpoint in the group that is the same as the category of the newly entered object point identified by the neural network, the score of the superpoint in the group is updated using the score of the newly entered object point. If the superpoint in the group falls within the minimum size of the category to which the newly entered object point belongs, the superpoint in the group will receive additional points.

[0024] According to an embodiment of the first aspect of this application, updating the semantic graph in response to the newly arrived object point includes:

[0025] If the minimum angular difference between a viewpoint corresponding to one of the object superpoints in the group and all viewpoints corresponding to the object superpoints in the group is greater than a predetermined angle, then update the historical data of the viewpoint information of any object superpoint in the group; and / or if the minimum dimensional difference between a viewpoint corresponding to one of the object superpoints in the group and all viewpoints corresponding to the object superpoints in the group exceeds a predetermined value, then update the historical data of the dimensional information of any object superpoint in the group.

[0026] According to an embodiment of the first aspect of this application, updating the semantic graph in response to the newly arrived object point includes:

[0027] If there are no object superpoints within the minimum size of the category to which the newly entered object point belongs, then the historical data of the newly entered object point is initialized, wherein only the current score, viewpoint information and size information of the newly entered object point will be recorded on the semantic graph.

[0028] According to an embodiment of the first aspect of this application, modifying the probability of the category to which the neural network identifies and the newly arrived object point belongs based on the updated semantic graph includes:

[0029] The probability of the category to which the newly entered object point belongs is corrected by using the updated score of the object superpoint in the group.

[0030] According to an embodiment in conjunction with the first aspect of this application, the probability of the category to which the newly arrived object point belongs is adjusted by considering the highest score among all object superpoints in the group that have the same category as the category of the newly arrived object point and the highest score among all object superpoints in the group that have other categories.

[0031] According to one embodiment of the first aspect of this application, the probability of the category to which the newly arrived object point belongs is corrected based on the sigmoid function.

[0032] In a second aspect of this application, an object detection system includes:

[0033] At least one memory for storing program instructions; and

[0034] At least one processor is configured to execute the program instructions, which cause the at least one processor to perform the following steps:

[0035] Based on the output from the neural network and augmented reality (AR), object representation data is created, wherein the object representation data includes object label information of objects identified by the neural network on the image, the three-dimensional position of object points, and viewpoint and size information of the object points from the AR frame;

[0036] Construct a semantic graph including object hyperpoints, wherein each object hyperpoint is represented by the historical score data of the object point, the viewpoint information, and the size information;

[0037] A set of object superpoints is determined in the semantic graph, wherein the positions of the object superpoints in the set are within a specific distance of the newly arrived object point, wherein the specific distance is determined based on the size of the object category identified by the neural network, and the newly arrived object point belongs to the category;

[0038] In response to the newly arrived object point, the semantic graph is updated, wherein the scores of object superpoints in the groups determined in the semantic graph are updated based on the information of the newly arrived object point; and

[0039] Based on the updated semantic graph, the probability of the neural network identifying the category to which the newly entered object point belongs is modified.

[0040] According to an embodiment of the second aspect of this application, updating the semantic graph in response to the newly arrived object point includes:

[0041] The score of the newly arrived object point is calculated by comparing its viewpoint information with the historical viewpoint information of the superpoints of the objects in the group and / or comparing its size information with the historical size information of the superpoints of the objects in the group. The newly arrived object point belongs to a category with a certain probability identified by the neural network, and the score of the newly arrived object point represents the change in the probability of the category to which the newly arrived object point belongs.

[0042] According to an embodiment of the second aspect of this application, updating the semantic graph in response to the newly arrived object point includes:

[0043] For the category of the superpoint in the group that is the same as the category of the newly entered object point identified by the neural network, the score of the superpoint in the group is updated using the score of the newly entered object point. If the superpoint in the group falls within the minimum size of the category to which the newly entered object point belongs, the superpoint in the group will receive additional points.

[0044] According to an embodiment of the second aspect of this application, modifying the probability of the category to which the neural network identifies and the newly arrived object point belongs based on the updated semantic graph includes:

[0045] The probability of the category to which the newly entered object point belongs is corrected by using the updated scores of the object superpoints in the group, wherein the highest score among all object superpoints in the group that have the same category as the newly entered object point and the highest score among all object superpoints in the group that have other categories are considered to correct the probability of the category to which the newly entered object point belongs.

[0046] In a third aspect of this application, a non-transitory computer-readable medium is provided having program instructions stored thereon, which, when executed by at least one processor, cause the at least one processor to perform any of the above-described object detection methods.

[0047] In this application, semantic graphs are used to improve the accuracy of object detection. Semantic points in the semantic graph are generated by combining object detection results from a neural network with pose data and 3D data points from an AR framework. The semantic graph consists of object hyperpoints and includes lists of scores corresponding to detected labels, lists of viewing directions, and lists of sizes. Based on this data, the probabilities from the neural network are modified. By modifying the probabilities of object labels or categories, the accuracy of object detection is improved. Attached Figure Description

[0048] To more clearly illustrate the embodiments of this application or related technologies, the accompanying drawings described in the embodiments are briefly described below. Obviously, these drawings represent only some embodiments of this application, and those skilled in the art can derive other drawings based on these drawings without incurring any cost.

[0049] Figure 1 A schematic diagram of the object detection architecture according to this application is shown.

[0050] Figure 2 This diagram shows a flowchart of the object detection method according to this application.

[0051] Figure 3 This shows a flowchart of the semantic graph update procedure according to this application.

[0052] Figure 4 This diagram shows a block diagram of the object detection system according to this application.

[0053] Figure 5 This diagram shows the block diagram of the update module of the object detection system according to this application.

[0054] Figure 6 A block diagram of an electronic device implementing the object detection method according to this application is shown. Detailed Implementation

[0055] The embodiments of this application will now be described in detail with reference to the accompanying drawings, focusing on their technical solutions, structural features, achieved objectives, and effects. Specifically, the terminology used in the embodiments of this application is only used to describe certain embodiments and is not intended to limit the invention.

[0056] The terms used in this application are defined as follows:

[0057]

[0058] Table 1

[0059] The following description uses DNN and AR frameworks as examples. However, other neural networks and similar augmented reality techniques can also be applied in this application, and the scope of this application is not limited to the examples given.

[0060] This application utilizes (3D) semantic graphs to improve the accuracy of 2D object detection DNNs. Semantic point clouds are generated by combining object detection results from the DNN with pose and 3D point results from an AR framework. The semantic graph consists of 3D hyperpoints, including lists of scores corresponding to detected labels, lists of viewing directions, and lists of sizes. Probabilities from the DNN are updated or modified based on this data. By modifying the probabilities of object labels or categories, the accuracy of object detection is improved.

[0061] In some embodiments of this application, the score calculation for the estimated object points considers not only the number of times a label has been detected at a given location, but also the detection based on viewing direction and size. This approach outperforms ordinary DNNs in AR scenes when the viewpoint changes continuously. This reduces the probability of false positives, i.e., when an object is identified as belonging to a category it has never recently seen from a similar viewing direction, but has been identified as another category multiple times at the same location. For example, this approach can improve false positives in situations where a bed is detected as a sofa in the current frame, but the semantic map shows that a bed has been consistently detected at the same location in previous frames. In another example, when the object detection DNN outputs a relatively low probability for an object label, but the semantic map indicates that the object at this location was viewed from a different direction some time ago, this approach increases the probability of that category. This approach improves the accuracy of 2D object detection within an AR framework without requiring additional training data to handle changes in the size and viewpoint of the task. This enables many AR applications, such as adding semantic labels to 3D point clouds, thereby triggering corresponding virtual content for the user.

[0062] Figure 1 A schematic diagram of the object detection architecture according to this application is shown. The object detection architecture of this application is described as follows.

[0063] In traditional AR sessions, when a device moves through a scene and returns to a previously visited location after any time interval, if the device is continuously and successfully tracked, the 6DoF camera pose (i.e., position and orientation) will remain almost identical to the previously recorded values. This means that the AR framework has the ability to remember a place. However, tracking can sometimes be lost when the AR framework's SLAM (VIO) fails due to a lack of sufficient feature matching between the current frame and some recent previous frames. In this case, relocalization can be used to re-estimate the device pose relative to the map by matching features in the input image with features stored in previously seen images. Another problem that can arise during VIO operation is that the device trajectory deviates from the true trajectory over time. To address this, intermittent relocalization is typically performed to detect revisited locations to break the loop. This ability of the AR framework to remember and recognize previously visited locations can be incorporated into the development of object detection methods and systems.

[0064] like Figure 1 As shown, the 3D semantic map is constructed from the outputs of the AR framework and the DNN. Object detection presents many challenges, such as variations in size and viewpoint, requiring the object detector to detect objects of different sizes in an image from different viewpoints. This challenge is overcome by using a category scale database to verify that the category of the detected object is consistent with the scale estimated by the AR framework. For example, an airplane should not appear in a 5m x 5m space. When the DNN detects an object, the viewpoint generated by the AR framework is stored. In some embodiments, consistent detections of the same object from different viewing directions and / or at different scales are preferred. In some embodiments, a probabilistic model is used to insert and update object category, viewpoint, and size information in the 3D semantic map. Figure 1 As shown, information is extracted from the 3D semantic graph to update the object label probabilities from the DNN.

[0065] Figure 2 A flowchart of the object detection method according to this application is shown. The object detection method is described in detail below.

[0066] In step S200, object representation data is created based on the output from the neural network and the AR framework.

[0067] In this step, object representation data is created using outputs from neural networks (e.g., DNNs) and AR frameworks (e.g., Apple's ARKit or Google's ARCore, which employ SLAM algorithms, more specifically VIO algorithms). Object representation data includes object label information (e.g., labels of objects recognized by the neural network on an image, such as chairs)... Figure 1The image (shown as a chair label) also includes the three-dimensional position of the object point and viewpoint and size information of the object point from the AR frame. The 2D object point of an object (e.g., a chair) in the image has a corresponding 3D object point estimated from the AR frame, where the 2D object point is generated by mapping the 3D object point onto the image.

[0068] In an illustrative example, for each frame, the DNN can output a list of N object categories, each with an associated bounding box and probability. For each object output, the object representation data structure is created as (loc, label, view, scale), where loc is the estimated 3D coordinates of an object point in the current frame, label is the object label from the DNN, view is the viewing direction (or viewpoint) from the camera to loc, and scale is the size information, which depends on the distance from the camera position to loc. For each frame, the AR framework generates 6 DoF camera poses and a sparse set of 3D points with global 3D coordinates, used to compute loc, view, and scale.

[0069] In step S202, a semantic graph including object hyperpoints is constructed.

[0070] In this step, each object superpoint is represented by the object point's historical score data, viewpoint information, and size information.

[0071] In one example instance, unlike the estimated object point data structure (loc, label, view, scale), the object superpoint is represented as (loc, list_score, list_view, list_scale). These three lists encode information from all previous frames in the AR session. 1) list_score(E1, E2, f3, ..., E l (…) Store the score E for each detected label l at that point. l The list of points; the higher the score, the higher the probability that the point belongs to category l. 2) list_view(v1, v2, v3, ...) stores a list of historical viewing directions (or viewpoints) from the camera position to the point when an object is detected; 3) list_scale(s1, s2, s3, ...) stores a list of historical dimensions from the camera position to the point when an object is detected. In an AR session, a point may be labeled with different categories at different times.

[0072] In one embodiment, the semantic map is constructed based on object points whose projections onto the image lie within the boundary regions of the objects identified by the neural network. That is, the semantic map is constructed by focusing only on 3D object points mapped to or falling within the boundary regions (e.g., bounding boxes) of the objects identified by the neural network. One reason for this is to reduce computation; another is that more accurate results can be obtained because only the information of interest is involved in the computation.

[0073] In one embodiment, object superpoints in the semantic map are constructed using the median point of an object point whose projection onto the image falls within the object's boundary region (e.g., a bounding box). The median point ensures that it falls within the object's boundary region on the image. Furthermore, this approach further reduces computational complexity.

[0074] In one illustrative example, to place objects detected by a DNN onto a 3D semantic map, some form of statistics can be performed on all 3D points whose projections onto the image fall within the 2D bounding boxes of the detected object labels. In one embodiment, the median value of all points in each of the XYZ dimensions is used to represent the object in the current view. This avoids assigning object labels to irrelevant points in the semantic map or the background. This makes the approach more robust and efficient. For example, as... Figure 1 As shown, the AR framework estimates the camera's pose and reconstructs some 3D points, such as dots (see...). Figure 1 (Right side). The median values ​​p(2.8733, 1.09483, 1.2345) of the dots within the 2D bounding box of the "chair" are used to represent the estimated object point loc. The viewing direction v(0.61497, 0.76871, 0.17458) is calculated from the distance from the camera position to loc. v is a normalized unit vector. The size information s is an integer defined as the floor of log2d, where d is the distance between the camera position and loc. For example, when the distance between the camera and loc is 1 meter, s = 0. Therefore, for the detected chair, a data structure (p, "Chair", v, s) is created to represent the detected object.

[0075] In step S204, a set of object superpoints in the semantic graph is determined, wherein the positions of the object superpoints in the set are within a specific distance of the newly entered object point.

[0076] In AR sessions, the viewpoint changes. For the same object, new object points are generated from different viewpoints. Furthermore, different labels or categories may be assigned to the same object. To fuse the new object point with object superpoints in the 3D semantic map, a set of object superpoints in the semantic map is first determined, where the superpoints are located within a specific distance of the new object point. More specifically, this specific distance is determined based on the size of the object category (e.g., chair category) recognized by the neural network, to which the new object point belongs. The size of the object category or label can be, for example, […]. Figure 1 The categories and sizes shown are obtained from the database.

[0077] In one embodiment, during the process of determining a set of object superpoints in a semantic graph, wherein the positions of the object superpoints in the set are within a specific distance of a newly arrived object point, the specific distance is the maximum size of the category to which the newly arrived object point belongs. For example, the size of the chair category is in the range of 0.5 meters to 1.5 meters, so the maximum size of the chair category is 1.5 meters.

[0078] In one illustrative instance, within an AR session, different estimated object points might be generated for the same object from different viewpoints because only a portion of the object's surface is visible in each image. Furthermore, the same object might be assigned different labels within the same session. To estimate the representation of each new object (loc... in label in view in scale in The data is then fused with the superpoints (loc, list_score, list_view, list_scale) in the semantic graph. First, the distance between the location loc in the semantic graph and the newly entered object point loc is calculated. in A set of superpoints within a specific distance. For this purpose, a minimum and maximum size are defined for each category (e.g., the size range for the chair category is 0.5 meters to 1.5 meters), and superpoints falling within the maximum size of a newly added object category are added to the set S. in It is processed in the middle.

[0079] In step S206, the semantic graph is updated in response to newly arrived object points.

[0080] In response to new object points in the AR session, the semantic map is updated. The updated semantic map is used to modify the probabilities of the new object's category and the category recognized by the neural network in subsequent processes, thereby improving object detection. When updating the semantic map, the scores of object superpoints in the groups identified in the semantic map are updated based on the information from the new object points. In other words, information from the new object points participates in the construction of the historical scores of that group of object superpoints in the semantic map.

[0081] Figure 3 This diagram displays a flowchart of the semantic graph updating procedure according to this application. The semantic graph updating in step S206 may include the following steps, namely steps 300 to 306.

[0082] In step 300, the score of the newly entered object point is calculated.

[0083] In one illustrative instance, a score is calculated for each new object point in category l. Here, two weights w are defined. v and w s , where 0≤w v w s ≤1. The calculation is as follows:

[0084]

[0085] Where, p l It is the probability of class l calculated by the DNN, and the weight w v and w s The calculation is as follows:

[0086]

[0087] w s =k s *s diff , if s diff <1 / k s ,orherwise w s =1

[0088] Among them, v diff It is the current viewing direction (or viewpoint) and S in The minimum angular difference between all viewing directions in a list_view containing all points. diff The higher the value, the higher the weight w. v The higher. Furthermore, when v diff Within 45 degrees, the weight w v Set to zero so that the semantic graph is updated only intermittently. And when v diff When the temperature is greater than 90 degrees, w v Set the upper limit to 1.

[0089] Similarly, s diff Is the current size and S in The minimum size difference between all dimensions in the list_scale of all points in the array. diff The higher the weight w s The higher. k s It is an s diffThe normalization factor. In one example, the size is defined as s = log2(d), where d is the distance between the camera position and the superpoint. Here, the size difference s needs to be calculated. diff Define a range such that the same object can be detected within this range. In this example, this range can be empirically chosen as [0, 5]. Therefore, k s Let's set it to 1 / 5 = 0.2.

[0090] In one embodiment, calculating the score of a newly arrived object point may include comparing the viewpoint information of the newly arrived object point with the historical viewpoint information of object superpoints in the group (e.g., v). diff ) and / or a comparison between the size information of the newly arrived object point and the historical size information of the object superpoints in the group (e.g., s diff The score of a new incoming object point is calculated by taking this into account. That is, the magnitude of the camera pose change can be estimated by comparing the viewpoint information of the new incoming object point with the historical viewpoint information of the superpoints in the group and / or by comparing the size information of the new incoming object point with the historical size information of the superpoints in the group. More specifically, the score of a new incoming object point represents the change in the probability of its class, as it belongs to a category identified by the neural network with a certain probability. A large change in camera pose will result in a higher score for the new incoming object point because, in this case, it is best to assign the new label identified by the neural network to the new incoming object point. Conversely, a small change in camera pose will result in a lower score for the new incoming object point because, in this case, it is best to maintain the already detected label for the new incoming object point.

[0091] In one embodiment, when calculating the score of a newly arrived object point, the score is calculated based on a first weight and a second weight, wherein the first weight is associated with the minimum angular difference between the viewpoint corresponding to the newly arrived object point and all viewpoints corresponding to object superpoints in the group, and the second weight is associated with the minimum dimensional difference between the size corresponding to the newly arrived object point and all sizes corresponding to object superpoints in the group. As shown in the example above, the first weight and the second weight can be w respectively. v and w s . Minimum angle difference v diff Used to determine the first weight, minimum size difference s diff Used to determine the second weight.

[0092] In one embodiment, if the minimum angle difference is less than a first predetermined angle, the first weight is set to a first value; and if the minimum angle difference is greater than a second predetermined angle, the first weight is set to a second value, where the first value is less than the second value and the first predetermined angle is less than the second predetermined angle. As shown in the example above, the first weight could be w. v If the minimum angular difference vdiff If the angle is less than 45 degrees, then the first weight w v Set to 0 if the minimum angle difference v diff If the angle is greater than 90 degrees, the first weight is set to 1.

[0093] In one embodiment, if the minimum size difference is within a predetermined range, the second weight is proportional to the minimum size difference; if the minimum size difference exceeds the predetermined range, the second weight is set to a fixed value. As shown in the example above, the second weight can be w. s If the minimum size difference s diff In 1 / k s Within this range, the second weight is related to w. s Minimum size difference s diff Proportional, if the minimum size difference s diff More than 1 / k s Then the second weight w s Set it to 1.

[0094] In one embodiment, the score of the newly arrived object point increases with increasing minimum angular difference and / or minimum size difference (e.g., With v diff and / or s diff (Increases with increasing); the score of the newly arrived object point decreases as the minimum angular difference and / or minimum size difference decreases (e.g., With v diff and / or s diff The decrease in minimum angle difference and / or minimum size difference indicates an increased chance of using the class probability of the new ingress point obtained from the neural network. In other words, a large change in camera pose will result in a higher score for the new ingress point, because in this case, it is best to assign the new label recognized by the neural network to the new ingress point. The decrease in minimum angle difference and / or minimum size difference indicates a decreased chance of using the class probability of the new ingress point obtained from the neural network. In other words, a small change in camera pose will result in a lower score for the new ingress point, because in this case, it is best to maintain the already detected label for the new ingress point.

[0095] In step 302, the scores of the superpoints of the objects in the group are updated using the scores of the newly entered object points.

[0096] In one example instance, given a score The behavior of new objects, for a neighboring superpoint n∈S in Its detection score will be updated for category l: Given the distance D between the newly entered point and n, its score The update is as follows:

[0097]

[0098] Here, for any superpoint within the neighborhood defined by the minimum size of the newly arrived object point, an additional value of 1 is added to the score of the detected label as a reward for being within the minimum object size of the estimated category.

[0099] In one embodiment, when updating the scores of object superpoints in the group, the scores of the object superpoints in the group are updated using the scores of the new object point for the same category as the new object point identified by the neural network. More specifically, the distance from the new object point to the object superpoint in the group with the same category is considered. For object superpoints falling between the maximum and minimum sizes of the new object point's category, their scores are updated by adding the new object point's score to their original scores. For object superpoints in the group with the same category as the new object point, if these object superpoints fall within the minimum size of the category to which the new object point belongs, these object superpoints receive an additional score (e.g., 1). This takes into account the number of times a label is detected at a certain location.

[0100] In step 304, the historical data of viewpoint information and / or size information are updated.

[0101] In one of the exemplary instances, if v diff If the angle is ≥45°, the viewing direction v is added to the list_view; if the group S in all points s diff If the value is ≥1, then the size s is added to list_scale, regardless of its label.

[0102] In one embodiment, if the minimum angular difference (e.g., v) between a viewpoint corresponding to one of the object hyperpoints in the group and all viewpoints corresponding to the object hyperpoints in the group is... diff ) greater than a predetermined angle (e.g., v) diff If the viewpoint is ≥45°, then update the historical data of the viewpoint information for any object superpoint in the group; and / or if the minimum size difference between the viewpoint corresponding to one of the object superpoints in the group and all viewpoints corresponding to the object superpoints in the group (e.g., s) is found. diff ) exceeds the predetermined value (e.g., s) diff If ≥1), then update the historical data of the size information of any object in the group that exceeds the limit.

[0103] In step 306, the historical data of the newly arrived object point is initialized under specific circumstances.

[0104] In one example instance, finally, if no superpoint exists within the minimum size distance of the newly arrived object point, all three lists—list_score, list_view, and list_scale—are reset to their respective values. v and s are initialized. For example, list_view(v) ​​and list_scale(s). Then, the new superpoints (loc, list_score, list_view, list_scale) are added to the semantic graph.

[0105] In one embodiment, if no object superpoint exists within the minimum size of the category to which the newly arrived object point belongs, the historical data of the newly arrived object point is initialized. This initialization means that only the current score, viewpoint information, and size information of the newly arrived object point are recorded on the semantic graph, and the previous or historical score, viewpoint information, and size information are initialized to zero or deleted.

[0106] In step S208, the probability of the category to which the neural network identifies and the newly entered object point belongs is modified based on the updated semantic graph.

[0107] In an exemplary instance, for each new object point with label l, its label probability p can be updated using superpoints stored in the semantic graph neighborhood. l First, find the highest score among all object superpoints with the label 'l'. Find the highest score among all objects with other labels. The probability p of a point p on the semantic graph map The modified sigmoid function is defined as follows:

[0108]

[0109] p map The minimum value is defined as 0.5 to ensure that this does not drastically reduce the output probability p of the DNN. l Therefore, the final probability of the object is:

[0110] p = p map *p l

[0111] For each frame, the final output is a list of bounding boxes, each with an output (l, p, bbox), where the label and bounding box are the same as the output of the DNN.

[0112] In one embodiment, the probability of a newly arrived object's category is corrected by using the updated scores of the object superpoints in the group. In another embodiment, the probability of a newly arrived object's category is corrected by considering the highest score among all object superpoints in the group that have the same category as the newly arrived object and the highest score among all object superpoints in the group that have other categories.

[0113] Figure 4 This shows a block diagram of the object detection system according to this application. (See diagram below.) Figure 4 As shown, an object detection system 40 is provided. The object detection system 40 includes a creation module 400, a construction module 402, a determination module 404, an update module 406, and a modification module 408.

[0114] The creation module 400 is configured to create object representation data based on the output from the neural network and augmented reality (AR), wherein the object representation data includes object label information of objects identified by the neural network on the image, the three-dimensional position of object points, and viewpoint and size information of the object points from the AR frame.

[0115] The construction module 402 is configured to construct a semantic graph including object superpoints, wherein each object superpoint is represented by the scoring history data of the object point, the viewpoint information, and the size information.

[0116] The determination module 404 is configured to determine a set of object superpoints in the semantic graph, wherein the positions of the object superpoints in the set are within a specific distance of a newly arrived object point, wherein the specific distance is determined based on the size of the object category identified by the neural network, to which the newly arrived object point belongs.

[0117] Update module 406 is configured to update the semantic graph in response to the newly arrived object point, wherein the scores of object superpoints in the group determined in the semantic graph are updated based on the information of the newly arrived object point.

[0118] The modification module 408 is configured to modify the probability of the neural network identifying the category to which the newly arrived object point belongs, based on the updated semantic graph.

[0119] Figure 5 This diagram shows the block diagram of the update module of the object detection system according to this application. Figure 5 As shown, the update module 406 of the object detection system 40 includes a calculation unit 500, a score update unit 502, a data update unit 504, and an initialization unit 506.

[0120] The calculation unit 500 is configured to calculate the score of the new object point by taking into account comparisons between the viewpoint information of the new object point and the historical viewpoint information of the object superpoints in the group and / or comparisons between the size information of the new object point and the historical size information of the object superpoints in the group, wherein the new object point belongs to a category with a certain probability identified by the neural network, and the score of the new object point represents the change in the probability of the category to which the new object point belongs.

[0121] The score update unit 502 is configured to update the score of the superpoints in the group with the score of the new object point for the category of the superpoints in the group that are the same as the category of the new object point identified by the neural network. If the superpoints in the group fall within the minimum size of the category to which the new object point belongs, the superpoints in the group will receive additional points.

[0122] The data update unit 504 is configured to update the historical data of the viewpoint information of any object superpoint in the group if the minimum angular difference between the viewpoint corresponding to one of the object superpoints in the group and all viewpoints corresponding to the object superpoints in the group is greater than a predetermined angle; and / or update the historical data of the size information of any object superpoint in the group if the minimum size difference between the viewpoint corresponding to one of the object superpoints in the group and all viewpoints corresponding to the object superpoints in the group exceeds a predetermined value.

[0123] The initialization unit 506 is configured to correct the probability of the category to which the newly entered object point belongs by using the updated scores of the object superpoints in the group, wherein the highest score among all object superpoints in the group that have the same category as the newly entered object point and the highest score among all object superpoints in the group that have other categories are considered when correcting the probability of the category to which the newly entered object point belongs.

[0124] For specific limitations regarding the object detection system, please refer to the limitations on the object detection method above, which will not be repeated here. The modules or units in the above object detection system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules or units can be embedded in hardware or independent of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module or unit.

[0125] The various modules or units in the object detection system can be implemented in the form of computer programs. These computer programs can run on a terminal or server. The program modules constituted by these computer programs can be stored in the memory of the terminal or server. When the computer program is executed by a processor, it implements the operations of the methods described in the embodiments of this application.

[0126] This application also provides a non-transitory computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform operations of an object detection method.

[0127] Figure 6 This is a block diagram of an electronic device 600 according to an embodiment of this application. For example, the electronic device 600 may be a mobile phone, a game controller, a tablet device, a medical device, a workout device, or a personal digital assistant (PDA).

[0128] Reference Figure 6 The electronic device 600 may include one or more of the following components: a housing 602, a processor 604, a memory 606, a circuit board 608, and a power supply circuit 610. The circuit board 608 is disposed within the space defined by the housing 602. The processor 604 and the memory 606 are disposed on the circuit board 608. The power supply circuit 610 is configured to supply power to each circuit or device of the electronic device 600. The memory 606 is configured to store executable program code. The processor 604 reads the executable program code stored in the memory 606 and runs a program corresponding to that executable program code to perform the object detection method of any of the above embodiments.

[0129] Processor 604 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processor 604 may include one or more processors 604 to execute instructions to perform actions in all or part of the steps in the methods described above. Furthermore, processor 604 may include one or more modules that facilitate interaction between processor 604 and other components. For example, processor 604 may include a multimedia module to facilitate interaction between multimedia components and processor 604.

[0130] Memory 606 is configured to store various types of data to support the operation of electronic device 600. Examples of such data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 606 can be implemented using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0131] The power supply circuit 610 supplies power to various components of the electronic device 600. The power supply circuit 610 may include a power management system, one or more power supplies, and any other components associated with the generation, management, and distribution of power to the electronic device 600.

[0132] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the methods described above.

[0133] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as instructions included in memory 606, which can be executed by processor 604 of electronic device 600 to perform the methods described above. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0134] Those skilled in the art will understand that the various units, modules, algorithms, and steps described and disclosed in the embodiments of this application are all implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions operate in hardware or software depends on the application conditions and the design requirements of the technical solution. Those skilled in the art can use different methods to implement the functions for each specific application, but such implementation should not exceed the scope of this application.

[0135] Those skilled in the art will understand that they can refer to the workflow of the systems, devices, and modules in the embodiments mentioned above, as the workflows of the systems, devices, and modules mentioned above are substantially the same. For ease of description and brevity, these workflows will not be described in detail.

[0136] It should be understood that the systems and methods disclosed in the embodiments of this application can be implemented in other ways. The embodiments described above are merely illustrative. The division of these modules is based solely on logical function, and other division methods exist in implementation. Multiple modules or components may be combined or integrated into another system. Some features may also be omitted or skipped. On the other hand, the mutual coupling, direct coupling, or communication coupling shown or discussed is performed through some ports, devices, or modules, whether indirectly or communicatively, by electrical, mechanical, or other means.

[0137] Modules used for interpretation as separate components may or may not be physically separated. The modules shown may or may not be physical modules, that is, located in one place or distributed across multiple network modules. Some or all of the modules are used according to the purposes of these embodiments.

[0138] Furthermore, each functional module in each embodiment can be integrated into a processing module that is physically independent, or integrated into a processing module that has two or more modules.

[0139] If software functional modules are implemented, used, and sold as products, they can be stored in a readable storage medium within a computer. Based on this understanding, the technical solutions proposed in this application can be implemented essentially or partially as software products. Alternatively, a portion of a technical solution beneficial to the prior art can be implemented as a software product. Software products in a computer are stored in a storage medium and include multiple commands for a computing device (e.g., a personal computer, server, or network device) to execute all or part of the steps disclosed in the embodiments of this application. This storage medium includes a USB flash drive, a portable hard drive, read-only memory (ROM), random access memory (RAM), a floppy disk, or other media capable of storing program code.

[0140] Although this application has been described in conjunction with what are considered to be the most practical and preferred embodiments, it should be understood that this application is not limited to the disclosed embodiments, but is intended to cover various arrangements made without departing from the broadest interpretation of the appended claims.

Claims

1. An object detection method, characterized in that, The method includes: The processor creates object representation data based on the output from the neural network and the augmented reality (AR) framework. The object representation data includes object label information of the objects identified by the neural network in the image, the three-dimensional position of the object points, and the viewpoint information and size information of the object points from the augmented reality framework. A semantic graph comprising object superpoints is constructed by a processor, wherein each object superpoint is represented by historical scoring data of the object point, a list of viewpoint information, and a list of size information. The historical scoring data includes a list of scores for each label detected at the object point. The list of viewpoint information includes historical viewing directions from the camera position to the object point when the object was detected. The list of size information includes a list of historical dimensions from the camera position to the object point when the object was detected. The semantic graph is constructed based on the object points, and the projection of the object points onto the image lies within the boundary region of the object identified by the neural network. The processor determines a set of object superpoints in the semantic graph, wherein the positions of the object superpoints in the set are within a specific distance of the newly entered object point, wherein the specific distance is determined based on the size of the object category identified by the neural network, and the newly entered object point belongs to the category; The semantic graph is updated by responding to the newly arrived object point, wherein the scores of object superpoints in the groups determined in the semantic graph are updated based on the information of the newly arrived object point; and The processor modifies the probability of the neural network identifying the category to which the newly arrived object point belongs based on the updated semantic map.

2. The object detection method according to claim 1, characterized in that, The object superpoints of the semantic graph are constructed using the projections onto the image at the midpoints of the object points within the boundary region of the object.

3. The object detection method according to claim 1, characterized in that, The specific distance is the maximum size of the category to which the newly arrived object point belongs.

4. The object detection method according to claim 1, characterized in that, The response to the newly arrived object point, updating the semantic graph includes: The score of the newly arrived object point is calculated by comparing its viewpoint information with the historical viewpoint information of the superpoints of the objects in the group and / or comparing its size information with the historical size information of the superpoints of the objects in the group. The newly arrived object point belongs to a category with a certain probability identified by the neural network, and the score of the newly arrived object point represents the change in the probability of the category to which the newly arrived object point belongs.

5. The object detection method according to claim 4, characterized in that, When calculating the score of the newly arrived object point, the score of the newly arrived object point is calculated according to a first weight and a second weight, wherein the first weight is associated with the minimum angular difference between the viewpoint corresponding to the newly arrived object point and all viewpoints corresponding to the object superpoint in the group, and the second weight is associated with the minimum size difference between the size corresponding to the newly arrived object point and all sizes corresponding to the object superpoint in the group.

6. The object detection method according to claim 5, characterized in that, If the minimum angle difference is less than the first predetermined angle, then the first weight is set to the first value; If the minimum angle difference is greater than the second predetermined angle, then the first weight is set to the second value, where the first value is less than the second value and the first predetermined angle is less than the second predetermined angle.

7. The object detection method according to claim 5, characterized in that, If the minimum size difference is within a predetermined range, the second weight is proportional to the minimum size difference; if the minimum size difference exceeds the predetermined range, the second weight is set to a fixed value.

8. The object detection method according to claim 5, characterized in that, The score of the newly entered object point increases as the minimum angle difference and / or the minimum size difference increases; The score of the new object point decreases as the minimum angle difference and / or the minimum size difference decreases, and an increase in the minimum angle difference and / or the minimum size difference indicates an increased chance of using the class probability of the new object point obtained from the neural network, while a decrease in the minimum angle difference and / or the minimum size difference indicates a decreased chance of using the class probability of the new object point obtained from the neural network.

9. The object detection method according to claim 1, characterized in that, The response to the newly arrived object point, updating the semantic graph includes: For the category of the superpoint in the group that is the same as the category of the newly entered object point identified by the neural network, the score of the superpoint in the group is updated using the score of the newly entered object point. If the superpoint in the group falls within the minimum size of the category to which the newly entered object point belongs, the superpoint in the group will receive additional points.

10. The object detection method according to claim 1, characterized in that, The response to the newly arrived object point, updating the semantic graph includes: If the minimum angular difference between a viewpoint corresponding to one of the object superpoints in the group and all viewpoints corresponding to the object superpoints in the group is greater than a predetermined angle, then update the historical data of the viewpoint information of any object superpoint in the group; and / or if the minimum dimensional difference between a viewpoint corresponding to one of the object superpoints in the group and all viewpoints corresponding to the object superpoints in the group exceeds a predetermined value, then update the historical data of the dimensional information of any object superpoint in the group.

11. The object detection method according to claim 1, characterized in that, The response to the newly arrived object point, updating the semantic graph includes: If there are no object superpoints within the minimum size of the category to which the newly entered object point belongs, then the historical data of the newly entered object point is initialized, wherein only the current score, viewpoint information and size information of the newly entered object point will be recorded on the semantic graph.

12. The object detection method according to claim 1, characterized in that, The step of modifying the probability of the category identified by the neural network and the category to which the newly arrived object point belongs based on the updated semantic graph includes: The probability of the category to which the newly entered object point belongs is corrected by using the updated score of the object superpoint in the group.

13. The object detection method according to claim 12, characterized in that, The probability of the category to which the new object point belongs is adjusted by considering the highest score among all object superpoints in the group that have the same category as the new object point and the highest score among all object superpoints in the group that have other categories.

14. The object detection method according to claim 12, characterized in that, The probability of the category to which the newly arrived object point belongs is corrected based on the sigmoid function.

15. An object detection system, characterized in that, The system includes: At least one memory for storing program instructions; and At least one processor is configured to execute the program instructions, which cause the at least one processor to perform the following steps: Based on the output from the neural network and the augmented reality (AR) framework, object representation data is created, wherein the object representation data includes object label information of the object identified by the neural network on the image, the three-dimensional position of the object point, and the viewpoint information and size information of the object point from the augmented reality framework; A semantic graph is constructed including object superpoints, wherein each object superpoint is represented by historical scoring data of the object point, a list of viewpoint information, and a list of size information. The historical scoring data includes a list of scores for each label detected at the object point. The list of viewpoint information includes historical viewing directions from the camera position to the object point when the object was detected. The list of size information includes a list of historical dimensions from the camera position to the object point when the object was detected. The semantic graph is constructed based on the object points, and the projection of the object points onto the image is located within the boundary region of the object recognized by the neural network. A set of object superpoints is determined in the semantic graph, wherein the positions of the object superpoints in the set are within a specific distance of a newly entered object point, wherein the specific distance is determined based on the size of the object category identified by the neural network, and the newly entered object point belongs to the category; In response to the newly arrived object point, the semantic graph is updated, wherein the scores of object superpoints in the groups determined in the semantic graph are updated based on the information of the newly arrived object point; and Based on the updated semantic graph, the probability of the neural network identifying the category to which the newly entered object point belongs is modified.

16. The object detection system according to claim 15, characterized in that, The response to the newly arrived object point, updating the semantic graph includes: The score of the newly arrived object point is calculated by comparing its viewpoint information with the historical viewpoint information of the superpoints of the objects in the group and / or comparing its size information with the historical size information of the superpoints of the objects in the group. The newly arrived object point belongs to a category with a certain probability identified by the neural network, and the score of the newly arrived object point represents the change in the probability of the category to which the newly arrived object point belongs.

17. The object detection system according to claim 15, characterized in that, The response to the newly arrived object point, updating the semantic graph includes: For the category of the superpoint in the group that is the same as the category of the newly entered object point identified by the neural network, the score of the superpoint in the group is updated using the score of the newly entered object point. If the superpoint in the group falls within the minimum size of the category to which the newly entered object point belongs, the superpoint in the group will receive additional points.

18. The object detection system according to claim 15, characterized in that, The step of modifying the probability of the category identified by the neural network and the category to which the newly arrived object point belongs based on the updated semantic map includes: The probability of the category to which the newly entered object point belongs is corrected by using the updated scores of the object superpoints in the group, wherein the highest score among all object superpoints in the group that have the same category as the newly entered object point and the highest score among all object superpoints in the group that have other categories are considered to correct the probability of the category to which the newly entered object point belongs.

19. A non-transitory computer-readable medium having stored program instructions thereon, which, when executed by at least one processor, cause the at least one processor to perform the object detection method according to claim 1.

Images