Object recognition device, object recognition method, and object recognition program

Abstract

This object recognition device comprises: an error distribution information collection unit that executes a trained object recognition AI in a simulation environment and accumulates a distribution of inference values from an inference result including confidence scores; and a likelihood calculation unit that calculates the likelihood of an inference result sequence in each of the position and orientation of an object from the distribution of the inference values accumulated for each of the position / orientation by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment.

Description

Technical Field

[0001] The present invention relates to an object recognition device, an object recognition method, and an object recognition program.

Background Art

[0002] One application example of artificial intelligence is object recognition in a space. In object recognition in a space, the position and orientation of a specific object (such as a human or a vehicle) are inferred by a bounding box for data acquired from an optical camera, LiDAR (Light Detection And Ranging), etc. Then, the inference result is utilized for controlling a work robot and monitoring work in a logistics warehouse or an outdoor civil engineering work site.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] Furthermore, the disclosures in the above-mentioned prior art documents are incorporated into this book by reference. The following analysis was conducted by the inventors.

[0005] Incidentally, training recognition models generally requires a large amount of annotated (bounding box) datasets for each domain, so a large amount of annotated (bounding box) datasets from real-world environments where the object recognition results will be used is necessary. However, obtaining a large amount of annotated (bounding box) datasets from real-world environments is often not easy, which becomes an obstacle to improving the accuracy of object recognition.

[0006] Furthermore, while object recognition results may contain errors, the results themselves do not provide information about the amount of error. For example, the estimated error can be used to determine whether the inference is accurate enough to estimate collision risk or control. Therefore, in the application of object recognition results, information about the reliability of the inference (estimated error) is needed in addition to the inferred value.

[0007] Object recognition results include not only position and orientation estimation but also a confidence score, although the correlation between the confidence score and the reliability of the inference result is weak. Also, the variability of the inference result itself (unstable inference results have low reliability) is useful as a reference, but stability is only a necessary condition for high accuracy (no variability does not necessarily mean high accuracy). For example, Non-Patent Literature 1 points out the discrepancy between the confidence score and the actual likelihood in image classification AI (Artificial Intelligence) and proposes a method to adjust the variability of the confidence score by applying a temperature parameter when applying softmax to the output (logit) of the NN as a way to correct the bias. Non-Patent Literature 2 concretizes and evaluates a method of calibration using multidimensional detection results (position, box size, object type) with the definition of likelihood as the probability that the inferred bounding box and ground truth are within a specific IOU threshold, and proposes calibration of the confidence score of a 2D object recognition AI. In regression tasks such as object recognition, the "degree of deviation" is inherently a continuous quantity. However, in Non-Patent Literature 2 for object detection tasks, the calibration is for binary inference of correct / incorrect answers, and additional data is required to estimate the parameters of the calibration function.

[0008] The object of the present invention is to provide an object recognition device, an object recognition method, and an object recognition program that contribute to providing an evaluation of the error (variability) in the inference value (object position and orientation confidence score) of an object recognition AI, in view of the above-mentioned problems. [Means for solving the problem]

[0009] In a first aspect of the present invention, an object recognition device is provided, comprising: an error distribution information collection unit that runs a trained object recognition AI in a simulation environment and accumulates a distribution of inference values ​​from inference results including a confidence score; and a likelihood calculation unit that calculates the likelihood of an inference result sequence at each position and orientation from the distribution of inference values ​​accumulated for each position and orientation of an object by the execution of the object recognition AI in the simulation environment and an inference result sequence of the object recognition AI in the real environment.

[0010] A second aspect of the present invention provides an object recognition method that involves running a trained object recognition AI in a simulation environment, accumulating a distribution of inference values ​​from the inference results including a confidence score, and calculating the likelihood of the inference result sequence at each position and orientation from the distribution of inference values ​​accumulated for each position and orientation of an object by the execution of the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment.

[0011] A third aspect of the present invention is provided, which provides an object recognition program that causes an information processing device to execute a process that runs a trained object recognition AI in a simulation environment, accumulates a distribution of inference values ​​from the inference results including the confidence score, and calculates the likelihood of the inference result sequence at each position and orientation from the distribution of inference values ​​accumulated for each position and orientation of an object by the execution of the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment. This program can be recorded on a computer-readable storage medium. The storage medium can be a non-transient medium such as semiconductor memory, hard disk, magnetic recording medium, or optical recording medium. The present invention can also be embodied as a computer program product. [Effects of the Invention]

[0012] According to each aspect of the present invention, it is possible to provide an object recognition device, an object recognition method, and an object recognition program that contribute to providing an evaluation of the error (variation) in the inference value (object position and orientation confidence score) of an object recognition AI. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a functional block diagram of an object recognition device according to the first embodiment. [Figure 2] Figure 2 shows an example of a sample space. [Figure 3] Figure 3 shows an example of inference values ​​for object recognition performed in the sample space. [Figure 4] Figure 4 shows an example of the distribution of inferred values ​​pi obtained in a real environment. [Figure 5] Figure 5 is a flowchart showing the procedure of the object recognition method according to the first embodiment. [Figure 6] Figure 6 is a functional block diagram of the object recognition device according to the second embodiment. [Figure 7] Figure 7 is a flowchart showing the procedure of the object recognition method according to the second embodiment. [Figure 8] Figure 8 shows an example of likelihood map visualization in the second embodiment. [Figure 9] Figure 9 shows an example of the hardware configuration of an object recognition device used in the embodiment. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the embodiments described below. In each drawing, the same or corresponding elements are appropriately denoted by the same reference numerals. Furthermore, it should be noted that the drawings are schematic, and the dimensional relationships and ratios of each element may differ from those of reality. Even between drawings, there may be parts where the dimensional relationships and ratios differ from each other.

[0015] (First Embodiment) FIG. 1 is a functional block diagram of an object recognition device according to the first embodiment. As shown in FIG. 1, the object recognition device 10 includes a simulation environment 11, an emulator 12, an object recognition AI 13, an error distribution information collection unit 14, an inference value data storage unit 15, a real environment sensor 16, and a likelihood calculation unit 17.

[0016] The simulation environment 11 is a virtual construction of the real environment to which the object recognition device 10 is applied. For example, each element of the spatial vector can be discretized and configured by its combination. Note that the approximation accuracy improves as the discretization of each element is refined, but it is preferable to appropriately limit the number of elements in consideration of the computational load.

[0017] The emulator 12 is a program that mimics the characteristics of the real environment sensor 16. The real environment sensor 16 may include errors or biases when detecting data in the real environment, and this error or bias may also cause an error in the inference result of the object recognition AI 13. The emulator 12 mimics the characteristics that cause the errors of the real environment sensor 16 so that information on the errors caused by the characteristics of the real environment sensor 16 can be obtained even when the object recognition AI 13 is executed in the simulation environment 11. Note that the emulator 12 can be implemented as a program separate from the simulation environment 11, but it can also be implemented as a part of the simulation environment 11.

[0018] The object recognition AI 13 is an artificial intelligence that estimates the position and orientation of an object existing in the space and the type of the object, etc., with detection data for the space as input. The detection data used for the input can be, for example, an image or a video, but the output of LiDAR (Light Detection And Ranging) technology that irradiates laser light and measures the distance to the target object and the shape of the target object, etc., based on the information of the reflected light can also be used.

[0019] The error distribution information collection unit 14 runs the trained object recognition AI 13 in the simulation environment 11 and estimates and stores the distribution of inferred values ​​from the inference results, including the confidence score. The inferred values ​​acquired by the error distribution information collection unit 14 are stored in the inferred value data storage unit 15 along with the true position and orientation of the object constructed in the simulation environment 11.

[0020] The real-world sensor 16 is a sensor that detects spatial information of the real environment in which the object recognition device 10 is applied. For example, it can be a device that combines optical equipment and an image sensor such as a CMOS, or a LiDAR device that irradiates laser light and measures the distance to an object and the shape of the object based on the information of the reflected light.

[0021] The likelihood calculation unit 17 calculates the likelihood of the inference result sequence for each position and orientation from the distribution of inference values ​​accumulated for each position and orientation of an object by the execution of the object recognition AI 13 in the simulation environment 11 and the inference result sequence of the object recognition AI 13 in the real environment. Likelihood is the plausibility of the values ​​of parameters such as the model or distribution that explain an observed value, and in this case, it means the plausibility of the position and orientation of the object that explains that inference value from the inference result of the object recognition AI 13. The inference result of the object recognition AI 13 may sometimes correctly infer the true position and orientation of the object, but due to the influence of errors contained in the data acquired by the real environment sensor 16 or the imperfections of the object recognition AI 13, it may also obtain inference values ​​that differ from the true position and orientation of the object. By calculating the likelihood, the likelihood calculation unit 17 calculates the most plausible position and orientation of the object that explains that inference value from the inference result of the object recognition AI 13.

[0022] Thus, since the likelihood of the object's position and orientation explaining the inference result from the object recognition AI 13 represents the most plausible position and orientation, the object recognition device 10 can provide error information for the object recognition AI 13's inference value (object position and orientation confidence score). In other words, by providing the object recognition AI 13's inference value with likelihood information, the object recognition device 10 can utilize the inference value while taking into account the influence of errors included in object recognition.

[0023] Here, we will explain in more detail how to calculate likelihood. (Definition of symbols) • p: An inference value vector containing position, orientation, and confidence level (eg[c, lx, ly, lz, dx, dy, dz, rz, s], where c is the object type, lx, ly, lz are the object positions, dx, dy, dz are the object sizes, rz is the rotation angle, and s is the confidence level.) • pi: The i-th (=1st, 2nd, ..., Nth) inference value output from the recognition model within a certain time period. g: True object position and orientation vector (eg[c, lx, ly, lz, dx, dy, dz, rz]) For simplicity, in the following explanation, we will assume that g = [lx, ly, rz] ∈ R. 3 • G: The set of g (ie.e., G = {gk | k = 1, 2, ..., K}) f(p|g): Probability density function of the inferred value p when the true object position is g.

[0024] (Object recognition in a simulation environment) 1. Define a sample space G which is the set of g. • Discretize each element of the vector and generate combinations of them. • Limiting the number of elements to ensure sampling efficiency and approximation accuracy. Figure 2 shows an example of a sample space. In the example of a sample space shown in Figure 2, g1=[1,1,0], g2=[1,2,0], g3=[-1,1,0], and g4=[-1,2,0] are defined.

[0025] 2. For each g in the simulation environment, execute the object recognition AI13 to calculate f(p|g). • The kernel density is estimated (or the multidimensional normal distribution is approximated) from the inferred value pi to obtain f(p|g). The obtained f(p|g) is essentially a sum of normal distributions, so it can be calculated quickly. Figure 3 shows examples of inference values ​​for object recognition performed in the sample space. Figure 3 shows the inference result f(p|g1) for g1 as a typical example of accurate inference, and the inference result f(p|g2) for g2 as a typical example of confidently incorrect inference. The inference result f(p|g1) for g1 accurately infers the true object position g1=[1,1,0], and the confidence level s is close to 1. In other words, this is an example where the inference result f(p|g1) can be trusted. On the other hand, the inference result f(p|g2) for g2 does not accurately infer the true object position g2=[1,2,0], yet the confidence level s is close to 1. In other words, this is an example where the confidence level s is high, but the inference result cannot be trusted.

[0026] 3. (optional) It is also possible to improve the estimation accuracy by calculating the inter-distribution distance between adjacent f(p|g) in the sample space, and if this distance exceeds a threshold, adding the intermediate point (gk+gl) / 2 to the sample space G and performing a resimulation.

[0027] (Calculating the likelihood of object recognition in a real-world environment) 4. Based on the inferred value pi obtained in the real environment and the probability density function f(p|g) obtained in the simulation environment, the likelihood distribution of the true object position g is approximately calculated. Figure 4 shows an example of the distribution of inferred values ​​pi obtained in the real environment. In the example shown in Figure 4, the inferred values ​​obtained in the real environment are p1=[1.1, 1.5, 0, 0.7], p2=[0.9, 1.2, 0, 0.9], and p3=[1.0, 1.1, 0, 0.8]. The likelihood of the true object position g is obtained by calculating the following joint probability using these inferred values ​​p1, p2, and p3 and the probability density function f(p|g) obtained in the simulation environment.

number

[0028] If, for example, the following likelihoods are obtained as a result of calculating the above joint probability, the likelihood of the true object position being g1 is 0.54, making it the most likely case. However, the likelihood of the true object position being g2 is also 0.44, making it also quite likely that the true object position is g2. In other words, it is possible to conclude that the true object position is most likely to be g1, but that there is likely to be a large error in the y-axis direction. By reflecting this error information in the control of operations in the real environment, it is possible to reduce control errors.

number

[0029] (Object recognition method) Figure 5 is a flowchart showing the procedure of the object recognition method according to the first embodiment. As shown in Figure 5, the object recognition method includes an error distribution information collection step (step S1) and a likelihood calculation step (step S2).

[0030] In the error distribution information collection step (step S1), the trained object recognition AI is executed in a simulation environment to accumulate the distribution of inferred values ​​from the inference results, including the confidence score. The configuration of the simulation environment and the method for calculating the distribution of inferred values ​​from the inference results can be the same as those used in the object recognition device 10.

[0031] In the likelihood calculation step (step S2), the likelihood of the inference result sequence at each position and orientation is calculated from the distribution of inference values ​​accumulated for each position and orientation of an object by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment. The method for calculating the likelihood can be the same as that used in the object recognition device 10.

[0032] (Second Embodiment) Figure 6 is a functional block diagram of the object recognition device according to the second embodiment. As shown in Figure 6, the object recognition device 20 includes a simulation environment 11, an emulator 12, an object recognition AI 13, an error distribution information collection unit 14, an inference value data storage unit 15, a sensor for the real environment 16, a likelihood calculation unit 17, and a likelihood map creation unit 18. The simulation environment 11, emulator 12, object recognition AI 13, error distribution information collection unit 14, inference value data storage unit 15, sensor for the real environment 16, and likelihood calculation unit 17 can be configured in the same way as the object recognition device 10 according to the first embodiment, so their explanation will be omitted here, and only the likelihood map creation unit 18 will be explained.

[0033] The likelihood map generation unit 18 visualizes the distribution of likelihoods on a map. As mentioned earlier, likelihood represents the most likely position and orientation of an object that explains an inferred value, so likelihood itself is already useful information for utilizing inferred values. However, for convenience, it is more convenient to visualize likelihoods on a map rather than outputting them as mere numerical values. Therefore, the likelihood map generation unit 18 can visualize the distribution of likelihoods using a Delaunay plot or a heatmap.

[0034] One method for visualizing the likelihood distribution using a Delaunay diagram is to create a Delaunay diagram for the sample space G (Delaunay triangulation), and then connect points with the same height using the line segments of the triangles as a set of points representing height (likelihood in this case), and project this onto the XY plane to obtain contour lines. Another method for visualizing the likelihood distribution using a heatmap is to define a grid that displays colors on the XY plane, and sample n points in order of proximity to the center point of one cell in the grid. Then, map the color from the colormap to the height of that cell, using the weighted average height (likelihood in this case) obtained by the distance from the center point.

[0035] (Object recognition method) Figure 7 is a flowchart showing the procedure of the object recognition method according to the second embodiment. As shown in Figure 7, the object recognition method includes an error distribution information collection step (step S1), a likelihood calculation step (step S2), and a likelihood map creation step (step S3).

[0036] In the error distribution information collection step (step S1), the trained object recognition AI is executed in a simulation environment to accumulate the distribution of inferred values ​​from the inference results, including the confidence score. The configuration of the simulation environment and the method for calculating the distribution of inferred values ​​from the inference results can be the same as those used in the object recognition device 10.

[0037] In the likelihood calculation step (step S2), the likelihood of the inference result sequence at each position and orientation is calculated from the distribution of inference values ​​accumulated for each position and orientation of an object by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment. The method for calculating the likelihood can be the same as that used in the object recognition device 10.

[0038] In the likelihood map creation step (step S3), the likelihood distribution is visualized on a map. As mentioned above, possible methods for visualizing the likelihood distribution on a map include using a Delaunay plot or using a heatmap. As previously stated, likelihood represents the most likely position and orientation of an object that explains the inferred value. However, for convenience, it is more convenient to visualize the likelihood on a map rather than outputting it as a simple numerical value. For example, as shown in Figure 8, visualizing the region of high-likelihood positions on a map makes it easy to understand that there is a large error in the y-axis direction. Note that Figure 8 is an example of likelihood map visualization in the second embodiment, and the method of visualizing the likelihood map can be appropriately selected depending on the application.

[0039] (Example hardware configuration) Figure 9 shows an example of the hardware configuration of an object recognition device used in the embodiment. That is, the object recognition devices 10 and 20 can realize each function of the object recognition devices 10 and 20 by executing the object recognition method described above as a program on an information processing device (computer) 30 that employs the hardware configuration shown in Figure 9. However, the hardware configuration example shown in Figure 9 is just one example of a hardware configuration that realizes each function of the object recognition devices 10 and 20, and is not intended to limit the hardware configuration of the object recognition devices 10 and 20. The object recognition devices 10 and 20 may include hardware not shown in Figure 9.

[0040] As shown in Figure 9, the hardware configuration that the object recognition devices 10 and 20 may employ includes, for example, a CPU (Central Processing Unit) 31, a main memory 32, an auxiliary memory 33, and an IF (Interface) unit 34, all interconnected by an internal bus.

[0041] The CPU 31 executes each instruction included in the object recognition program run by the information processing device (computer) 30. The main memory 32 is, for example, RAM (Random Access Memory) and temporarily stores various programs, such as the object recognition program run by the information processing device (computer) 30, for processing by the CPU 31.

[0042] The auxiliary storage device 33 is, for example, an HDD (Hard Disk Drive) and is capable of storing various programs, such as object recognition programs, executed by the information processing device (computer) 30, for medium to long term. These various programs, such as object recognition programs, can be provided as program products recorded on a non-transitory computer-readable storage medium.

[0043] The IF unit 34 provides, for example, an interface for the input and output of object recognition devices 10 and 20.

[0044] The information processing device (computer) 30, which employs the hardware configuration described above, realizes the functions of the object recognition devices 10 and 20 by executing the aforementioned object recognition method as a program.

[0045] Some or all of the above embodiments may also be described as follows, but are not limited to the following: [Note 1] An error distribution information collection unit that runs a pre-trained object recognition AI in a simulation environment and accumulates the distribution of inference values ​​from the inference results, including the confidence score, A likelihood calculation unit calculates the likelihood of the inference result sequence for each position and orientation from the distribution of inference values ​​accumulated for each position and orientation of an object by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment. An object recognition device equipped with the following features. [Note 2] The object recognition device according to Appendix 1, comprising a likelihood map creation unit for visualizing the distribution of the aforementioned likelihoods on a map. [Note 3] The likelihood map creation unit is an object recognition device as described in Appendix 2, which visualizes the distribution of the likelihood using a Delaunay diagram. [Note 4] The likelihood map creation unit is an object recognition device as described in Appendix 2, which visualizes the distribution of the likelihood using a heat map. [Note 5] The object recognition device according to any one of the appendices 1 to 4, wherein the error distribution information collection unit acquires data of the simulation environment via an emulator that mimics the characteristics of a sensor used for object recognition in the actual environment. [Note 6] The object recognition device according to any one of the appendices 1 to 5, wherein the likelihood calculation unit calculates the likelihood by calculating the joint probability from the inference result sequence for each position and orientation. [Note 7] An object recognition device according to any one of the appendices 1 to 6, which utilizes the results of the likelihood mentioned above for controlling operations in the actual environment. [Note 8] The trained object recognition AI is run in a simulation environment, and the distribution of inference values ​​is accumulated from the inference results, including the confidence score. An object recognition method that calculates the likelihood of the inference result sequence for each position and orientation of an object from the distribution of inference values ​​accumulated for each position and orientation of an object by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment. [Note 9] The object recognition method described in Appendix 8, which visualizes the distribution of the aforementioned likelihoods on a map. [Note 10] The trained object recognition AI is run in a simulation environment, and the distribution of inference values ​​is accumulated from the inference results, including the confidence score. An object recognition program that causes an information processing device to perform a process of calculating the likelihood of the inference result sequence at each position and orientation from the distribution of inference values ​​accumulated for each position and orientation of an object by executing the object recognition AI in the aforementioned simulation environment and the inference result sequence of the object recognition AI in the actual environment.

[0046] Furthermore, the disclosures of the above-mentioned patent documents and other materials cited are incorporated into this document by reference. Within the framework of the full disclosure of the present invention (including the claims), further modifications and adjustments to the embodiments or examples are possible based on the fundamental technical concept. Also, within the framework of the full disclosure of the present invention, various combinations or selections (including partial deletions) of various disclosure elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible. In other words, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that a person skilled in the art could make in accordance with the technical concept. In particular, with respect to the numerical ranges described in this document, any numerical value or sub-range included within that range should be interpreted as being specifically described, even if not otherwise stated. Furthermore, the disclosures of the above-mentioned cited documents may, if necessary, be used in part or in whole as part of the disclosure of the present invention, in accordance with the spirit of the present invention, and these may also be considered to be included in the disclosures of this application. [Explanation of symbols]

[0047] 10,20 Object recognition device 11 Simulation Environment 12 Emulators 13 Object recognition AI 14 Error Distribution Information Collection Unit 15. Inference Value Data Storage Unit 16 Sensors for real-world applications 17 Likelihood Calculation Unit 18. Likelihood Map Creation Section 30 Information Processing Devices 31 CPU 32 Main storage 33 Auxiliary storage device 34 IF section

Claims

1. An error distribution information collection unit that runs a pre-trained object recognition AI in a simulation environment and accumulates the distribution of inferred values ​​from the inference results, including the confidence score, A likelihood calculation unit calculates the likelihood of the inference result sequence for each position and orientation from the distribution of inference values ​​accumulated for each position and orientation of an object by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment. An object recognition device equipped with the following features.

2. The object recognition device according to claim 1, further comprising a likelihood map creation unit for visualizing the distribution of the likelihoods on a map.

3. The object recognition device according to claim 2, wherein the likelihood map creation unit visualizes the distribution of the likelihood using a Delaunay diagram.

4. The object recognition device according to claim 2, wherein the likelihood map creation unit visualizes the distribution of the likelihood using a heat map.

5. The object recognition device according to claim 1, wherein the error distribution information collection unit acquires data of the simulation environment via an emulator that mimics the characteristics of a sensor used for object recognition in the actual environment.

6. The object recognition device according to claim 1, wherein the likelihood calculation unit calculates the likelihood by calculating a joint probability from the inference result sequence for each position and orientation.

7. The object recognition device according to any one of claims 1 to 6, wherein the result of the likelihood is used to control the work in the actual environment.

8. The trained object recognition AI is run in a simulation environment, and the distribution of inferred values ​​is accumulated from the inference results, including the confidence score. An object recognition method that calculates the likelihood of the inference result sequence for each position and orientation of an object from the distribution of inference values ​​accumulated for each position and orientation of an object by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment.

9. The object recognition method according to claim 8, wherein the distribution of the likelihood is visualized on a map.

10. The trained object recognition AI is run in a simulation environment, and the distribution of inferred values ​​is accumulated from the inference results, including the confidence score. An object recognition program that causes an information processing device to perform a process of calculating the likelihood of the inference result sequence at each position and orientation from the distribution of inference values ​​accumulated for each position and orientation of an object by executing the object recognition AI in the simulation environment and the inference result sequence of the object recognition AI in the real environment.

Images