Object recognition device, object recognition method, and object recognition program

Abstract

This object recognition device comprises: an error distribution information collection unit that executes a trained object recognition AI in a simulation environment and accumulates a distribution of inference values from an inference result including confidence scores; and a likelihood calculation unit that calculates the likelihood of an inference result sequence in each of the position and orientation of an object from the distribution of the inference values accumulated for each of the position / orientation by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment.

Description

OBJECT RECOGNITION DEVICE, OBJECT RECOGNITION METHOD, AND OBJECT RECOGNITION PROGRAM

[0001] The present invention relates to an object recognition device, an object recognition method, and an object recognition program.

[0002] One application of artificial intelligence is spatial object recognition. In spatial object recognition, the position and orientation of a specific object (such as a person or vehicle) is inferred using a bounding box based on data acquired from optical cameras and LiDAR (Light Detection and Ranging). The results of this inference are then used to control work robots and monitor their work in logistics warehouses and outdoor civil engineering work sites.

[0003] Guo, Chuan, et al. "On calibration of modern neural networks." International conference on machine learning. PMLR, 2017. Kueppers, Fabian, et al. "Multivariate confidence calibration for object detection." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition workshops. 2020.

[0004] The disclosures of the above-mentioned prior art documents are incorporated herein by reference. The following analysis has been carried out by the present inventors.

[0005] Training a recognition model generally requires a large amount of annotated (bounding box) datasets for each domain, which in turn requires a large amount of annotated (bounding box) datasets from the actual environment in which the object recognition results are to be utilized. However, obtaining a large amount of annotated (bounding box) datasets from the real environment is often not easy, which becomes an obstacle to improving the accuracy of object recognition.

[0006] Furthermore, although the results of object recognition may contain errors, the results themselves do not provide information on the amount of error. For example, the error probability can be used to determine whether the inference is likely to have the accuracy required for collision risk estimation and control. Therefore, in the application of object recognition results, information on the reliability of the inference (error probability) as well as the inference value is required.

[0007] Object recognition results include not only position and orientation estimates but also a confidence score, but the correlation between the confidence score and the reliability of the inference result is weak. Furthermore, while the variability of the inference result itself (unstable inference results have low reliability) is also useful, stability is merely a necessary condition for high accuracy (no variability necessarily implies high accuracy). For example, Non-Patent Document 1 points out the discrepancy between the confidence score and the actual likelihood in image classification AI (artificial intelligence) and proposes a method to correct the bias by adjusting the variability of the confidence score by applying a temperature parameter when applying softmax to the NN output (logit). Furthermore, Non-Patent Document 2 specifies and evaluates a calibration method using multidimensional detection results (position, box size, object type) to define likelihood as the probability that the inferred bounding box and ground truth are within a specific IOU threshold, and proposes calibration of the confidence score of 2D object recognition AI. In addition, in regression tasks such as object recognition, the "degree of deviation" is essentially a continuous quantity, but in the object detection task in Non-Patent Document 2, calibration is performed for binary inference of correct / incorrect, and additional data is required to estimate the parameters of the calibration function.

[0008] In view of the above-mentioned problems, an object of the present invention is to provide an object recognition device, an object recognition method, and an object recognition program that contribute to providing an evaluation of the error (variation) in the inference value (object position / posture confidence score) of object recognition AI.

[0009] In a first aspect of the present invention, an object recognition device is provided that includes an error distribution information collection unit that runs a trained object recognition AI in a simulation environment and accumulates a distribution of inference values ​​from inference results including a confidence score, and a likelihood calculation unit that calculates the likelihood of an inference result sequence at each position and orientation from the distribution of inference values ​​accumulated for each object position and orientation by running the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment.

[0010] In a second aspect of the present invention, an object recognition method is provided, which executes a trained object recognition AI in a simulation environment, accumulates a distribution of inference values ​​from inference results including a confidence score, and calculates the likelihood of the inference result sequence at each position and orientation from the distribution of inference values ​​accumulated for each object position and orientation by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment.

[0011] A third aspect of the present invention provides an object recognition program that causes an information processing device to execute the following process: execute a trained object recognition AI in a simulation environment, accumulate a distribution of inference values ​​from inference results including a confidence score, and calculate the likelihood of the inference result sequence at each position and orientation from the distribution of inference values ​​accumulated for each object position and orientation by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment. This program can be recorded on a computer-readable storage medium. The storage medium can be a non-transient medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. The present invention can also be embodied as a computer program product.

[0012] According to each aspect of the present invention, it is possible to provide an object recognition device, an object recognition method, and an object recognition program that contribute to providing an evaluation of the error (variation) in the inferred value (object position / posture confidence score) of object recognition AI.

[0013] FIG. 1 is a functional block diagram of an object recognition device according to a first embodiment. FIG. 2 is a diagram showing an example of a sample space. FIG. 3 is a diagram showing an example of inferred values ​​of object recognition performed in a sample space. FIG. 4 is a diagram showing an example of the distribution of inferred values ​​p obtained in a real environment. FIG. 5 is a flowchart showing the steps of an object recognition method according to the first embodiment. FIG. 6 is a functional block diagram of an object recognition device according to a second embodiment. FIG. 7 is a flowchart showing the steps of an object recognition method according to the second embodiment. FIG. 8 is a diagram showing an example of visualization of a likelihood map in the second embodiment. FIG. 9 is a diagram showing an example of the hardware configuration of an object recognition device used in the embodiments.

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the embodiments described below. In addition, the same or corresponding elements in each drawing are appropriately designated by the same reference numerals. Furthermore, it should be noted that the drawings are schematic, and the dimensional relationships and ratios of each element may differ from those in reality. There may also be parts in which the dimensional relationships and ratios differ between the drawings.

[0015] 1 is a functional block diagram of an object recognition device according to Embodiment 1. As shown in Fig. 1, the object recognition device 10 includes a simulation environment 11, an emulator 12, an object recognition AI 13, an error distribution information collection unit 14, an inference value data storage unit 15, a real-world sensor 16, and a likelihood calculation unit 17.

[0016] The simulation environment 11 is a virtual representation of the real environment to which the object recognition device 10 is applied, and can be configured by, for example, discretizing each element of a spatial vector and combining them. Note that the more finely each element is discretized, the higher the approximation accuracy will be, but it is preferable to appropriately limit the number of elements in consideration of the calculation load.

[0017] The emulator 12 is a program that mimics the characteristics of the real environment sensor 16. The real environment sensor 16 may contain errors or biases when detecting data from the real environment, and these errors or biases can cause errors in the inference results of the object recognition AI 13. The emulator 12 mimics the characteristics of the real environment sensor 16 that cause errors, so that information on errors caused by the characteristics of the real environment sensor 16 can be obtained even when the object recognition AI 13 is executed in the simulation environment 11. The emulator 12 can be implemented as a program separate from the simulation environment 11, or can be implemented as part of the simulation environment 11.

[0018] The object recognition AI 13 is an artificial intelligence that uses detection data for a space as input to estimate the position, posture, and type of object present in that space. The detection data used for input can be, for example, an image or video, but it can also use the output of LiDAR (Light Detection and Ranging) technology, which irradiates a laser beam and measures the distance to the object, the shape of the object, etc. based on the information of the reflected light.

[0019] The error distribution information collection unit 14 executes the trained object recognition AI 13 in the simulation environment 11, and estimates and stores the distribution of inference values ​​from the inference results including the confidence score. The inference values ​​acquired by the error distribution information collection unit 14 are stored in the inference value data storage unit 15 together with the true positions and orientations of objects constructed in the simulation environment 11.

[0020] The real environment sensor 16 is a sensor that detects spatial information of the real environment to which the object recognition device 10 is applied, and can be, for example, a device that combines an optical device with an imaging element such as a CMOS, or a LiDAR device that emits laser light and measures the distance to an object and the shape of the object based on information from the reflected light.

[0021] The likelihood calculation unit 17 calculates the likelihood of the inference result sequence for each position and orientation from the distribution of inference values ​​accumulated for each object position and orientation by the execution of the object recognition AI 13 in the simulation environment 11 and the inference result sequence of the object recognition AI 13 in the real environment. Likelihood refers to the likelihood of the value of a parameter, such as a model or distribution, that explains a given observed value. In this case, it refers to the likelihood of the object's position and orientation that explains the inference value from the inference result of the object recognition AI 13. The inference result of the object recognition AI 13 may correctly infer the true position and orientation of the object, but may also obtain inference values ​​that differ from the true position and orientation of the object due to the influence of errors contained in the data acquired by the real environment sensor 16 or imperfections in the object recognition AI 13. By calculating the likelihood, the likelihood calculation unit 17 calculates the most likely position and orientation of the object that explains the inference value from the inference result of the object recognition AI 13.

[0022] In this way, since likelihood means the plausible position and orientation of the object position and orientation that explain the inference result of the object recognition AI 13, the object recognition device 10 can provide error information for the inference value (object position / orientation confidence score) of the object recognition AI 13. In other words, by adding likelihood information to the inference value of the object recognition AI 13 and providing it, the object recognition device 10 can utilize the inference value taking into account the influence of errors included in the object recognition.

[0023] Here, we will explain in more detail how to calculate likelihood. (Definition of symbols) ・p: Inference value vector including position / orientation and confidence level (e.g., [c, lx, ly, lz, dx, dy, dz, rz, s], where c is the object type, lx, ly, lz are the object position, dx, dy, dz are the object size, rz is the rotation angle, and s is the confidence level.) ・pi: The i (= 1, 2, .., N)th inference value output from the recognition model at a certain time ・g: True object position / orientation vector (e.g., [c, lx, ly, lz, dx, dy, dz, rz]) For simplicity in the following explanation, g = [lx, ly, rz] ∈ R 3・G: Set of g (i.e., G={gk | k=1,2,...K}) ・f(p|g): Probability density function of the inferred value p when the true object position is g

[0024] (Object recognition in a simulation environment) 1. Define a sample space G, which is a set of g. - Discretize each element of the vector and generate their combinations - Restrict elements to ensure sampling efficiency and approximation accuracy Figure 2 is a diagram showing an example of a sample space. In the example sample space shown in Figure 2, g1 = [1,1,0], g2 = [1,2,0], g3 = [-1,1,0], g4 = [-1,2,0] are defined.

[0025] 2. Execute the object recognition AI 13 for each g in the simulation environment to calculate f(p|g). - Obtain f(p|g) from the inferred value p using kernel density estimation (or multidimensional normal distribution approximation). - The obtained f(p|g) is basically a distribution of the sum of normal distributions, allowing for fast calculations. Figure 3 shows an example of inferred values ​​from object recognition performed in a sample space. Figure 3 shows the inference result f(p|g1) for g1 as a typical example of accurate inference, and the inference result f(p|g2) for g2 as a typical example of a confident error. The inference result f(p|g1) for g1 accurately infers the true object position g1 = [1,1,0], and the confidence level s is close to 1. In other words, this is an example where the inference result f(p|g1) can be trusted. On the other hand, the inference result f(p|g2) for g2 does not accurately infer the true object position g2 = [1,2,0], but the confidence level s is close to 1. In other words, this is an example where the confidence level s is high but the inference result cannot be trusted.

[0026] 3. (Optional) Calculate the inter-distribution distance between adjacent f(p|g) in the sample space, and if this distance exceeds a threshold, add the midpoint (gk+gl) / 2 to the sample space G and perform a re-simulation to improve the estimation accuracy.

[0027] (Likelihood calculation for object recognition in a real environment) 4. The distribution of the likelihood of the true object position g is approximately calculated based on the inferred values ​​p obtained in the real environment and the probability density function f(p|g) obtained in the simulation environment. Figure 4 shows an example of the distribution of inferred values ​​p obtained in the real environment. In the example shown in Figure 4, the inferred values ​​obtained in the real environment are p1 = [1.1, 1.5, 0, 0.7], p2 = [0.9, 1.2, 0, 0.9], and p3 = [1.0, 1.1, 0, 0.8]. The likelihood of the true object position g is obtained by calculating the following joint probability using these inferred values ​​p1, p2, and p3 and the probability density function f(p|g) obtained in the simulation environment.

[0028] For example, if the following likelihood is obtained as a result of calculating the above joint probability, the likelihood that the true object position is g1 is 0.54, making it the most likely, but the likelihood that the true object position is g2 is 0.44, making it also likely that the true object position is g2. In other words, it can be determined that the true object position is likely to be g1, but that there will be a large error in the y-axis direction. By reflecting this error information in the control of tasks in a real environment, it is possible to reduce control errors.

[0029] (Object Recognition Method) Fig. 5 is a flowchart showing the steps of the object recognition method according to the first embodiment. As shown in Fig. 5, the object recognition method includes an error distribution information collection step (step S1) and a likelihood calculation step (step S2).

[0030] In the error distribution information collection step (step S1), the trained object recognition AI is executed in a simulation environment, and the distribution of inference values ​​is accumulated from the inference results, including the confidence score. The configuration of the simulation environment and the method of calculating the distribution of inference values ​​from the inference results can be the same as those of the object recognition device 10.

[0031] In the likelihood calculation step (step S2), the likelihood of the inference result sequence at each position and orientation is calculated from the distribution of inference values ​​accumulated for each position and orientation of the object by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment. The likelihood calculation method can be the same as that of the object recognition device 10.

[0032] Second Embodiment Fig. 6 is a functional block diagram of an object recognition device according to a second embodiment. As shown in Fig. 6, an object recognition device 20 includes a simulation environment 11, an emulator 12, an object recognition AI 13, an error distribution information collection unit 14, an inference value data storage unit 15, a real environment sensor 16, a likelihood calculation unit 17, and a likelihood map creation unit 18. The simulation environment 11, the emulator 12, the object recognition AI 13, the error distribution information collection unit 14, the inference value data storage unit 15, the real environment sensor 16, and the likelihood calculation unit 17 can be configured in the same way as the object recognition device 10 according to the first embodiment, so their description will be omitted here, and only the likelihood map creation unit 18 will be described.

[0033] The likelihood map creation unit 18 visualizes the likelihood distribution in a map. As described above, likelihood means the plausible position and orientation of the object that explains the inferred value, and thus the likelihood itself is already information that contributes to utilizing the inferred value. However, considering convenience, it is more convenient to visualize the likelihood in a map rather than simply outputting it as a numerical value. Therefore, it is conceivable that the likelihood map creation unit 18 visualizes the likelihood distribution using a Delaunay diagram or a heat map.

[0034] The method for visualizing the distribution of likelihood using a Delaunay diagram is to create a Delaunay diagram for the sample space G (Dlaunay triangulation), and connect points with the same height with line segments, which are treated as a set of points representing height (likelihood in this case), and project this onto the XY plane to create contour lines. The method for visualizing the distribution of likelihood using a heat map is to define a grid that displays colors on the XY plane, and sample n points in order of proximity to the center point of a single cell in the grid. Then, the height (likelihood in this case), weighted by the distance from the center point, is used as the height of that cell, and colors are mapped from the color map.

[0035] (Object Recognition Method) Fig. 7 is a flowchart showing the steps of an object recognition method according to the second embodiment. As shown in Fig. 7, the object recognition method includes an error distribution information collection step (step S1), a likelihood calculation step (step S2), and a likelihood map creation step (step S3).

[0036] In the error distribution information collection step (step S1), the trained object recognition AI is executed in a simulation environment, and the distribution of inference values ​​is accumulated from the inference results, including the confidence score. The configuration of the simulation environment and the method of calculating the distribution of inference values ​​from the inference results can be the same as those of the object recognition device 10.

[0037] In the likelihood calculation step (step S2), the likelihood of the inference result sequence at each position and orientation is calculated from the distribution of inference values ​​accumulated for each position and orientation of the object by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment. The likelihood calculation method can be the same as that of the object recognition device 10.

[0038] In the likelihood map creation step (step S3), the likelihood distribution is visualized on a map. As described above, the likelihood distribution can be visualized using a Delaunay diagram or a heat map. As mentioned above, likelihood refers to the plausible position and orientation of the object that explains the inference value. However, for convenience, it is more convenient to visualize the likelihood on a map rather than simply outputting it as a numerical value. For example, as shown in FIG. 8, when regions with high likelihood are visualized on a map, it is easy to see that there is a large error in the y-axis direction. Note that FIG. 8 is a diagram showing an example of visualization of a likelihood map in the second embodiment, and the method for visualizing the likelihood map can be selected appropriately depending on the application.

[0039] (Hardware Configuration Example) Fig. 9 is a diagram showing an example of the hardware configuration of an object recognition device used in the embodiment. That is, the object recognition devices 10 and 20 enable each function of the object recognition devices 10 and 20 to be realized by executing the above-described object recognition method as a program on an information processing device (computer) 30 employing the hardware configuration shown in Fig. 9. However, the hardware configuration example shown in Fig. 9 is an example of a hardware configuration that realizes each function of the object recognition devices 10 and 20, and is not intended to limit the hardware configuration of the object recognition devices 10 and 20. The object recognition devices 10 and 20 may include hardware not shown in Fig. 9.

[0040] As shown in FIG. 9, the hardware configuration that can be adopted by the object recognition devices 10 and 20 includes a CPU (Central Processing Unit) 31, a main memory device 32, an auxiliary memory device 33, and an IF (Interface) unit 34, which are interconnected by, for example, an internal bus.

[0041] The CPU 31 executes each command included in the object recognition program executed by the information processing device (computer) 30. The main storage device 32 is, for example, a RAM (Random Access Memory), and temporarily stores various programs, such as the object recognition program executed by the information processing device (computer) 30, for processing by the CPU 31.

[0042] The auxiliary storage device 33 is, for example, a hard disk drive (HDD), and is capable of storing, for the medium to long term, various programs such as an object recognition program executed by the information processing device (computer) 30. Various programs such as an object recognition program can be provided as a program product recorded on a non-transitory computer-readable storage medium.

[0043] The IF unit 34 provides an interface related to input and output of the object recognition devices 10 and 20, for example.

[0044] The information processing device (computer) 30 employing the above-described hardware configuration implements the functions of the object recognition devices 10 and 20 by executing the above-described object recognition method as a program.

[0045] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. [Supplementary Note 1] An object recognition device comprising: an error distribution information collection unit that executes a trained object recognition AI in a simulation environment and accumulates a distribution of inference values ​​from inference results including a confidence score; and a likelihood calculation unit that calculates the likelihood of an inference result sequence at each position and orientation from the distribution of inference values ​​accumulated for each position and orientation of an object by executing the object recognition AI in the simulation environment and an inference result sequence of the object recognition AI in a real environment. [Supplementary Note 2] The object recognition device according to Supplementary Note 1, comprising a likelihood map creation unit that visualizes the likelihood distribution in a map. [Supplementary Note 3] The object recognition device according to Supplementary Note 2, wherein the likelihood map creation unit visualizes the likelihood distribution using a Delaunay diagram. [Supplementary Note 4] The object recognition device according to Supplementary Note 2, wherein the likelihood map creation unit visualizes the likelihood distribution using a heat map. [Supplementary Note 5] The object recognition device according to any one of Supplements 1 to 4, wherein the error distribution information collection unit acquires data from the simulation environment via an emulator that mimics the characteristics of a sensor used for object recognition in the real environment. [Supplementary Note 6] The object recognition device according to any one of Supplements 1 to 5, wherein the likelihood calculation unit calculates the likelihood by calculating a joint probability from an inference result sequence at each position and orientation. [Supplementary Note 7] The object recognition device according to any one of Supplements 1 to 6, wherein the likelihood result is utilized to control an operation in the real environment. [Supplementary Note 8] An object recognition method comprising: executing a trained object recognition AI in a simulation environment, accumulating a distribution of inference values ​​from inference results including a confidence score, and calculating the likelihood of the inference result sequence at each position and orientation from the distribution of inference values ​​accumulated for each object position and orientation by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment. [Supplementary Note 9] The object recognition method according to Supplementary Note 8, wherein the likelihood distribution is visualized on a map.[Appendix 10] An object recognition program that causes an information processing device to execute the following process: execute a trained object recognition AI in a simulation environment, accumulate a distribution of inference values ​​from inference results including a confidence score, and calculate the likelihood of the inference result sequence at each position and orientation from the distribution of inference values ​​accumulated for each object position and orientation by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment.

[0046] The disclosures of the above-cited patent documents and other documents are incorporated herein by reference. Modifications and adjustments of the embodiments and examples are possible within the scope of the entire disclosure of the present invention (including the claims), and further based on the basic technical concepts thereof. Furthermore, various combinations and selections (including partial deletions) of various disclosed elements (including elements of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the scope of the entire disclosure of the present invention. In other words, the present invention naturally embraces various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure and technical concepts, including the claims. In particular, with regard to the numerical ranges described herein, any numerical value or subrange within the range should be construed as specifically described, even if not otherwise specified. Furthermore, the disclosures of the above-cited documents, when used in part or in whole in combination with the disclosures herein as part of the disclosure of the present invention, in accordance with the spirit of the present invention, are also deemed to be included in the disclosures of this application.

[0047] 10, 20 Object recognition device 11 Simulation environment 12 Emulator 13 Object recognition AI 14 Error distribution information collection unit 15 Inference value data storage unit 16 Real environment sensor 17 Likelihood calculation unit 18 Likelihood map creation unit 30 Information processing device 31 CPU 32 Main memory device 33 Auxiliary memory device 34 IF unit

Claims

1. An error distribution information collection unit that executes the trained object recognition AI in a simulation environment and accumulates the distribution of inference values ​​from inference results including confidence scores; a likelihood calculation unit that calculates the likelihood of an inference result sequence at each position and posture from a distribution of inference values ​​accumulated for each position and posture of an object by executing the object recognition AI in the simulation environment and an inference result sequence of the object recognition AI in the real environment; An object recognition device comprising:

2. The object recognition device according to claim 1 , further comprising a likelihood map creation unit that visualizes the likelihood distribution on a map.

3. The object recognition device according to claim 2 , wherein the likelihood map creation unit visualizes the distribution of the likelihood using a Delaunay diagram.

4. The object recognition device according to claim 2 , wherein the likelihood map creation unit visualizes the distribution of the likelihood using a heat map.

5. The object recognition device according to claim 1 , wherein the error distribution information collecting unit acquires the data of the simulation environment via an emulator that imitates the characteristics of a sensor used for object recognition in the real environment.

6. The object recognition device according to claim 1 , wherein the likelihood calculation unit calculates the likelihood by calculating a joint probability from a sequence of inference results at each position and orientation.

7. 7. The object recognition device according to claim 1, wherein the likelihood result is utilized to control an operation in the real environment.

8. The trained object recognition AI is executed in a simulation environment, and a distribution of inference values ​​is accumulated from the inference results, including the confidence score. An object recognition method that calculates the likelihood of an inference result sequence at each position and posture from the distribution of inference values ​​accumulated for each object position and posture by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment.

9. The object recognition method according to claim 8 , wherein the likelihood distribution is visualized in a map.

10. The trained object recognition AI is executed in a simulation environment, and a distribution of inference values ​​is accumulated from the inference results, including the confidence score. An object recognition program that causes an information processing device to execute a process that calculates the likelihood of an inference result sequence at each position and posture from the distribution of inference values ​​accumulated for each position and posture of an object by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment.