Recognition processing method and recognition processing device

Abstract

To make it possible to determine a parameter for article recognition processing for many types of articles using a small amount of teaching data.SOLUTION: A recognition processing device includes: a storage unit 410 that stores a plurality of patterns containing a plurality of registered articles and each having a representative parameter value; a reception unit 401 that receives at least a model of a new article and information on the new article having a plurality of scene images; a recognition processing unit 402 that performs recognition processing on the plurality of scene images of the new article by using a representative parameter of each of the plurality of patterns; a determination unit 404 that selects, as the most similar pattern of the new article, a pattern corresponding to the representative parameter value determined to be the best as a result of the recognition processing by the recognition processing unit 402; and an output unit 409 that outputs the representative parameter value as a recognizable parameter value of the new article when the result of the recognition processing on the new article using the representative parameter value of the selected most similar pattern is good.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a recognition processing method and a recognition processing device, and more particularly to a technique for determining parameters in the recognition processing of a wide variety of objects. [Background technology]

[0002] A picking robot is an intelligent robot that has a robot arm with a gripper at the end and sorts items and palletizes / depalletizes cases based on information obtained by sensors, and is used in logistics warehouses, for example. The performance of a picking robot depends on the object recognition process used to analyze the information obtained by the sensors.

[0003] A technology that extracts features from image data acquired by a sensor and estimates the position of the object to be recognized in the image data based on the features is widely used as an object recognition process. In relation to the design of the features, there are generally several adjustable parameters that affect the recognition performance. Appropriate parameter values ​​for a group of objects to be recognized can be automatically calculated by a computer if a model of the group of objects to be recognized and image data (hereinafter referred to as teaching data) in which the object is photographed and the object position is known for each object are prepared. However, the model and teaching data must be created and prepared manually. In particular, a large number of teaching data are required for each object to ensure the robustness of the calculated parameter values. Therefore, images must be taken of each object and object position information must be added, which requires a lot of manpower and labor costs. For example, in the case of a logistics warehouse, the number of objects to be recognized and parameters must be determined is enormous, so it is necessary to perform parameter setting work efficiently.

[0004] In order to efficiently perform parameter setting work for many kinds of objects, it is conceivable to classify the object group into several classes and set appropriate parameter values ​​for each class. For example, Patent Document 1 discloses a parameter setting method that identifies the class into which the object to be recognized is classified, and determines the parameter values ​​corresponding to the class as appropriate parameter values ​​for the object to be recognized, thereby enabling highly accurate recognition. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-68008 A Summary of the Invention [Problem to be solved by the invention]

[0006] The parameter setting described in Patent Document 1 assumes highly abstract patterns such as people and automobiles. Therefore, it is relatively easy to determine what patterns to prepare and how to classify the recognition target. However, when there is a large amount of diversity within the same pattern, it becomes difficult to achieve high recognition performance with the same parameter values. For example, in the case of cardboard boxes handled in a logistics warehouse, the sizes and patterns of the boxes vary widely, so it is necessary to further subdivide the patterns. However, as the pattern subdivision progresses, appropriate pattern classification becomes far removed from human senses, and it becomes non-obvious what patterns to prepare and how to classify the items. In particular, when determining appropriate parameter values ​​for a new item, in order to utilize pattern information, it is important to appropriately determine which pattern the item is classified into.

[0007] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to enable the determination of parameters for article recognition processing for many types of articles using a small amount of teaching data. [Means for solving the problem]

[0008] A preferred example of the recognition processing method according to the present invention is a recognition processing method in which a computer executes an object recognition process, comprising the steps of: storing a plurality of patterns including a plurality of registered articles, each pattern having a representative parameter value, in a storage unit; receiving new item information having at least a model of the new item and a plurality of scene images; The representative parameter of each of the plurality of patterns value A recognition processing step of performing a recognition process on a plurality of scene images of the novel article using the above-mentioned method; selecting the pattern corresponding to the representative parameter value determined to be the best as a result of the recognition process as the most similar pattern to the new article; and if the result of the recognition processing of the new article using the representative parameter value of the selected most similar pattern is good, determining the representative parameter value as a recognizable parameter value of the new article.

[0009] A preferred embodiment of the recognition processing device according to the present invention includes a storage unit that stores a plurality of patterns including a plurality of registered articles, each of which has a representative parameter value; A receiving unit that receives at least a model of a new item and information on the new item having a plurality of scene images; The representative parameter of each of the plurality of patterns value A recognition processing unit that performs recognition processing of a plurality of scene images of the novel item using a determination unit that selects the pattern corresponding to the representative parameter value that is determined to be the best as a result of the recognition processing by the recognition processing unit as the most similar pattern of the new article; and an output unit that outputs the representative parameter value as a recognizable parameter value of the new article if the result of the recognition process of the new article using the representative parameter value of the selected most similar pattern is good. The present invention can also be understood as a program for causing a computer to execute the above recognition processing method. Effect of the Invention

[0010] According to the present invention, it is possible to determine parameters for the article recognition process for many types of articles using a small amount of teaching data. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram showing an overview of an article recognition process according to the first embodiment. [Diagram 2] FIG. 13 is a diagram illustrating an example of parameter calculation. [Diagram 3] FIG. 13 is a diagram showing an example of pattern classification of a group of items. [Figure 4] FIG. 2 is a diagram showing functional blocks of the item recognition processing apparatus. [Diagram 5] FIG. 11 is a diagram showing a processing operation of parameter determination in the article recognition processing device. [Figure 6] 11A and 11B are diagrams illustrating an example of a recognition process using representative parameters. [Figure 7] 11A and 11B are diagrams illustrating an example of determining a result of a recognition process using a representative parameter. [Figure 8] FIG. 13 is a diagram showing an example of parameter determination based on a representative parameter value of a most similar pattern. [Figure 9] 11A and 11B are diagrams illustrating an example of a method for adding item position information to a scene image. [Figure 10] FIG. 13 illustrates an example of joint optimization. [Figure 11] FIG. 13 is a diagram illustrating an example of parameter determination based on the result of joint optimization. [Figure 12] FIG. 13 is a diagram illustrating an example of parameter determination based on a result of individual optimization. [Figure 13] FIG. 2 is a diagram illustrating an example of a hardware configuration of an item recognition processing apparatus. [Figure 14] FIG. 11 is a diagram illustrating a processing operation of parameter determination according to the second embodiment. [Figure 15] FIG. 11 is a diagram showing an example of parameter determination and pattern update based on a representative parameter value of a most similar pattern according to the second embodiment. [Figure 16]FIG. 11 is a diagram showing an example of parameter determination and pattern update based on the result of joint optimization according to the second embodiment. [Figure 17] FIG. 11 is a diagram showing an example of parameter determination and pattern update based on the result of individual optimization according to the second embodiment. [Figure 18] FIG. 11 is a diagram illustrating a processing operation of parameter determination according to the third embodiment. [Figure 19] FIG. 11 is a perspective view showing an application example of an article recognition processing apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. In describing the embodiments, the same reference numerals will be used to designate the same components in the various drawings, and duplicated descriptions will be omitted. EXAMPLES

[0013] The process of determining parameters in the article recognition processing apparatus according to the first embodiment will be described.

[0014] FIG. 1 shows an overview of an article recognition process realized by computer processing. The computer executes the recognition process using a model of the object to be recognized (hereinafter simply referred to as the object), a scene image including the object captured by an image capturing means such as a camera, and parameter values ​​of the recognition process as input data, and outputs estimated position information of the object in the scene image as output data. The processing procedure of the recognition process and details of each data depend on the recognition method used. The object model is data that characterizes the object, and includes CAD data of the object and feature amounts extracted from the image of the object. The scene image is an image having a format such as an RGB image or a depth image, or a combination thereof. A scene image generally includes one or more object objects, but there is no problem in the recognition process even if the object object is not included.

[0015] The parameter value θ of the recognition process is a value that determines the performance of the recognition process, such as the recognition rate and speed, and is generally expressed as a multidimensional vector quantity. The estimated position information that is output can be a set of pixel values ​​of the image if you are interested in the position of the object on a 2D image, or a set of coordinate values ​​in 3D space if you are interested in the position and orientation of the object in 3D space. Although these are numerical data, they can also be expressed on a scene image as shown in Figure 1.

[0016] If the scene image has item position information, it is possible to judge whether the recognition result is a success or failure by comparing the item position information with the output estimated position information. Evaluation indices include IoU (Intersection Over Union) if you are interested in the item position on a 2D image, and mean square error if you are interested in the item position and orientation in 3D space, and the value of the evaluation index is compared with a threshold value.

[0017] On the other hand, when the scene image does not have the item position information, if the estimated position information can be displayed on a display device so as to be visible to humans, the operator can judge the success or failure by looking at the display. In this case, the item position information of the scene image is not necessary, so that manpower and work costs can be saved.

[0018] Here, an example of a recognition method is described. To obtain the 3D position and orientation of a target object contained in a scene image, a set of an RGB image and a depth image is used as the scene image. To create a model of the target object, first, SIFT feature points and SIFT feature quantities are calculated from 2D images obtained by photographing the target object from various angles. SIFT (Scale-Invariant Feature Transform) is a type of image feature quantity called a local feature quantity, and SIFT feature points are expressed as pixel positions in a 2D image, and SIFT feature quantities are expressed as real vector quantities. A CAD model of the target object is prepared, and the SIFT feature quantities are linked to data points on the CAD model that correspond to the obtained SIFT feature points, and these are used as the model of the target object.

[0019] When the computer executes the recognition process, SIFT feature points and SIFT feature quantities are calculated from the RGB scene image, and three-dimensional coordinate values ​​corresponding to those SIFT feature points are obtained from the depth scene image. Then, the correspondence between the three-dimensional feature points of the model and the scene image is obtained based on the feature quantity values, and a transformation matrix connecting the two is calculated from the correspondence. The transformation matrix corresponds to the above-mentioned estimated position information. In this recognition method, parameters related to the calculation of SIFT feature points and feature quantities, parameters related to the calculation of a transformation matrix from the correspondence of three-dimensional feature points, etc. become parameters of the recognition process. Note that the processing procedure and effects of this embodiment are not limited to the above-mentioned recognition method.

[0020] FIG. 2 shows an example of calculating appropriate parameter values ​​for the recognition process. The computer executes an optimization calculation based on input data including a model of the target item, multiple scene images including the target item, and item position information on the scene images, and calculates parameter values ​​for the recognition process as output data. The optimization calculation is executed according to a processing procedure of an optimization method using parameter values ​​that maximize an evaluation function that returns a value according to the parameter value. This makes it possible to obtain parameter values ​​according to the purpose through the design of the evaluation function. For example, if the evaluation function is the recognition rate for multiple input scene images, the obtained parameter values ​​are expected to maximize the recognition rate for the multiple input scene images. Furthermore, by adding a term that depends on the recognition processing speed, it is possible to calculate parameter values ​​that enable the fastest possible recognition processing.

[0021] In the following description, an evaluation function is assumed to obtain parameter values ​​that will maximize the recognition rate for multiple input scene images. In this case, it should be noted that, in general, the greater the variety of input scene images, the wider the range of application of the obtained parameter values ​​(the higher the robustness). In other words, in order to obtain robust parameter values, a large number of scene images (training data) with object position information attached are required, resulting in significant manpower and labor costs. As will be described later, this embodiment includes a processing procedure that enables the determination of highly robust parameters even with a small amount of training data.

[0022] Here, an example of an optimization method will be described. Examples of the optimization method include grid search and Bayesian optimization. In grid search, for example, the parameter space is divided into grids of an appropriate size, and an evaluation function value is calculated at a parameter value corresponding to each grid point, and a parameter value that is the maximum is selected. In Bayesian optimization, a parameter value that is expected to obtain a larger evaluation function value is searched for based on already calculated parameter values ​​and the evaluation function value under those values. In optimizing parameters for recognition processing, recognition processing using a certain parameter value is often performed to calculate an evaluation function value, and the calculation cost is relatively high. Therefore, an optimization method that is as efficient as possible is preferable. Note that the processing procedure and effects of this embodiment are not limited to the optimization method described above.

[0023] In the example of Fig. 2, the input data is for one type of target object, but two or more types of objects may be used. In the example described below, the above process is used to obtain parameter values ​​that enable simultaneous recognition of groups of different types of objects.

[0024] FIG. 3 shows an example of pattern classification of a group of items. In FIG. 3, the horizontal axis represents each item pattern in a plurality of groups of items, and the vertical axis represents the parameter value. The vertical bar above each item represents the range of parameter values ​​that realizes the requirement recognition performance for each item. Here, for simplicity, one-dimensional parameters are assumed, but two-dimensional or more parameters can be considered in the same way. Please note that the items shown are examples for easy understanding of the explanation, and do not accurately reflect the actual ones. The parameter values ​​that satisfy the requirement recognition performance change depending on the requirement recognition performance, but generally have a finite range. Therefore, there are cases where the requirement recognition performance can be achieved with the same parameter values ​​even for different items. In this embodiment, focusing on this point, a group of items in which the same parameter values ​​that can achieve the requirement recognition performance can be found is considered as one pattern. In the example of FIG. 3, five items are classified into two pattern groups, pattern A consisting of three items and pattern B consisting of two items, by pattern classification. Here, the same parameter value found in each pattern is called a representative parameter value. This embodiment proposes a specific use of this pattern for determining an appropriate parameter value for an item.

[0025] To further explain the above pattern classification, the pattern classification in this embodiment does not generally match human senses. In other words, it can be said that this pattern classification performs pattern classification that is convenient for the purpose of parameter determination, based on the criterion that a group of objects for which there are parameter values ​​that simultaneously achieve the required recognition performance are classified as the same pattern. Furthermore, due to this clear criterion, as will be described later, it is possible to appropriately determine which pattern an object for which new parameters are to be set is classified into. Next, this pattern classification is not unique. It will become clear later that it is important to keep the number of patterns as small as possible, but the fact that it is not unique does not eliminate the effect of this embodiment. Furthermore, even if the pattern classification is the same, there is a degree of freedom in the selection of representative parameter values. This also does not eliminate the effect of this embodiment.

[0026] FIG. 4 shows functional blocks of the article recognition processing device. The reception unit 401 receives input of information on new items (e.g., model information and scene images). The reception unit may also be called an input unit. The recognition processing unit 402 performs item recognition processing on the information input by the reception unit 401. The target item, scene image, and parameter value change depending on the transition source. The representative parameter value evaluation unit 403 evaluates the performance of the representative parameter value for the new item. For each representative parameter value, recognition processing is executed and a recognition rate is calculated based on the result of the recognition success / failure signal reception unit.

[0027] The most similar pattern determination unit 404 compares the recognition rates between the representative parameter values ​​corresponding to the multiple patterns, and determines the pattern corresponding to the representative parameter value with the highest recognition rate as the most similar pattern. If the recognition rate is equal to or higher than a required value, it outputs the result to the optimal parameter value output unit 409, and if the recognition rate is lower than the required value, it transitions to the annotation signal receiving unit 406.

[0028] The recognition success / failure signal receiving unit 405 receives a signal indicating whether the recognition result by the recognition processing unit 402 is successful or unsuccessful. In one example, an image of the recognition result is displayed on a display device, and an operator looks at the image to determine whether the recognition result is successful or unsuccessful, and the result of the determination is received from the input unit.

[0029] The annotation signal receiving unit 406 receives an annotation signal for a part of the failure scene of the new item with the representative parameter value of the most similar pattern. Then, when the processing result of the simultaneous optimization by the optimization unit 407 is less than the required value and transition is made from the recognition performance evaluation unit 408, the correct scene of the new item is also included as a target. If there is a failure scene of the new item to which correct answer information has not yet been added at this stage, the annotation signal receiving unit 406 can receive an annotation signal for a part of the failure scene. Here, the inventors mean that simultaneous optimization means, when the recognition rate of the new item does not satisfy the required performance, selecting the most similar pattern for the new item, adding the new item to the most similar pattern, and replacing (updating) the representative parameters (details will be described later).

[0030] The optimization unit 407 performs simultaneous optimization using the scene image of the new article to which the correct answer information is added by the annotation signal receiving unit 406 and information on the most similar pattern. Alternatively, the optimization unit 407 performs individual optimization based on the information on the new article. The calculated parameter values ​​are evaluated by the recognition performance evaluation unit 408. Here, the inventors mean that individual optimization means creating a new pattern that satisfies required performance based on the information on the new article (details will be described later).

[0031] The recognition performance evaluation unit 408 evaluates the recognition rate for the new article of the parameter values ​​calculated by the optimization unit 407, and checks whether it satisfies the required value. If the result of the joint optimization does not satisfy the required value, the process returns to the optimization unit 407 and individual optimization is performed.

[0032] The output unit 409 outputs the calculated optimal parameter value of the new object, and displays it, for example, on a display device. If the value is less than the required value, it is determined that recognition is impossible, and the process returns to the optimization unit 407. The pattern information DB 410 is a storage unit that stores information on registered object groups (e.g., model information and scene images) and pattern information extracted from them. The hardware configuration of the object recognition processing device will be described later with reference to FIG. 13.

[0033] Fig. 5 is a flowchart of the parameter determination processing operation in the item recognition processing device. The parameters of a new item are determined by the processing procedure shown in Fig. 5. In this embodiment, in order to execute the item recognition processing, input information including at least a model of the new item and a scene image is required. In the following description, it is assumed that parameters of one or more items have been determined and that pattern classification of the items is performed based on that information.

[0034] First, the recognition processing unit 402 recognizes the new item from the scene image of the new item using each of the representative parameter values ​​of the multiple patterns for the input information received by the receiving unit 401 (S05). Based on the recognition result of the recognition processing unit 402 and in response to a signal from the recognition success / failure signal receiving unit 405, the representative parameter evaluation unit 403 judges the recognition result for each representative parameter value and selects the representative parameter value that is considered to be the best (S10). Note that the pattern corresponding to the representative parameter value is called the most similar pattern.

[0035] Here, the processing of steps S05-S10 will be specifically described with reference to FIG. 6. It is assumed that the recognition processing of the new article is performed for each scene image of the new article using each representative parameter value of a plurality of patterns A, B, etc., and output data of estimated position information as shown in FIG. 6 is obtained. For ease of understanding, each estimated position information is shown on the corresponding scene image. Note that the recognition processing is performed for all scene images of the prepared new article, but the recognition processing may be limited to only a part of them as necessary. Whether the recognition is successful or unsuccessful is judged based on each estimated position information, and an evaluation of the recognition performance of each representative parameter value (i.e., the recognition rate) is obtained. The evaluations for each pattern are compared, and the one with the best evaluation is selected as the most similar pattern. In the example of FIG. 6, pattern A with a recognition rate of 0.95 is selected as the most similar pattern.

[0036] Here, if the scene image of the new article is provided with article position information, the representative parameter evaluation unit 403 can automatically determine whether the recognition has been successful or not by using an appropriately set threshold value. However, at this stage, the article position information is not necessarily required for the scene image of the new article. Therefore, in this embodiment, for example, a determination as shown in FIG. 7 is adopted. That is, each piece of estimated position information is expressed on the corresponding scene image and displayed on the display device. The operator looks at the display and determines whether the recognition has been successful or not, and inputs the determination result (success or failure) from the input unit.

[0037] For scene images of new items for which the recognition result is determined to be successful, the output estimated position information is stored in the storage unit and can be used thereafter as the item position information for that scene image. This process makes it possible to automatically determine the result of the recognition of new items for that scene image. Note that there is no guarantee that the item position information at this point is optimal, so the following process is important.

[0038] 5, next, the most similar pattern determination unit 404 determines whether the result of the recognition using the representative parameter value of the most similar pattern determined in step S10 is good (S20). This can be determined, for example, by comparing with a preset threshold value.

[0039] If it is judged to be good in step S20 (S20: YES), the parameter values ​​capable of recognizing the new item are determined as the representative parameter values ​​of the most similar pattern, and output to the output unit 409 (S21). In this case, it is not necessary to calculate appropriate parameter values ​​for the new item by the process shown in FIG. 2, and the parameter values ​​can be determined without adding item position information to the scene image of the new item. FIG. 8 shows an example of this process, in which pattern A is selected as the most similar pattern to the new item. As shown in FIG. 8, the representative parameter value of pattern A is a value included in both the parameter range that satisfies the required recognition performance of the item group of pattern A and the parameter range that satisfies the required recognition performance of the new item.

[0040] At this stage, even if the subsequent processing in Fig. 5 is not performed, there is an advantage that parameters can be determined with a small amount of teaching data. The subsequent processing can further enhance this advantage.

[0041] If it is determined in step S20 that the result is not satisfactory (S20: NO), the annotation signal receiving unit 406 assigns item position information to a part of the scene image of the new item that was determined to have failed to be recognized in the recognition process using the representative parameter value of the most similar pattern (S30). For example, FIG. 9 shows an example of assigning item position information to a scene image. As shown in FIG. 9, the scene image is displayed on a display device, and the operator can input appropriate item position information from the input unit while viewing the display. For example, when considering a bounding box on the scene image, it is only necessary to select four points on the scene image.

[0042] Next, the optimization unit 407 calculates parameters using the scene image of the new item and the most similar pattern (S40). For example, as shown in FIG. 10, in addition to the scene image to which the model of the new item and the item position information are added, information on the most similar pattern is used as input data. Specifically, the information on the most similar pattern includes the model and scene image of the item classified into the most similar pattern, the item position information added thereto, intermediate data obtained when the parameters of the item are determined, and the representative parameter value of the most similar pattern. At this point, since there are generally few scene images to which the item position information of the new item is added that can be used for parameter calculation, information on the most similar pattern with the closest properties is used to compensate for this. As a result, even if there is little teaching data for the new item, it is expected that the calculated parameter value will be such that the new item can be sufficiently recognized. It is considered that the higher the similarity between the new item and the most similar pattern (for example, the higher the recognition rate of the new item using the representative parameter value of the most similar pattern obtained in step S10), the higher the possibility. Hereinafter, this parameter calculation is referred to as simultaneous optimization.

[0043] Here, some concrete examples of simultaneous optimization are described. The first example is an example of mixing of teaching data. In the optimization process assumed in this example, a parameter value that increases the recognition performance of the input scene image as much as possible is searched for. The teaching data of the item classified as the most similar pattern is considered to contain information for calculating the representative parameter value of the most similar pattern. On the other hand, the scene image of the new item whose recognition result using the representative parameter value is judged to be successful is considered to contain at least part of the information for calculating the representative parameter value if appropriate item position information is given. In other words, even if the teaching data of the item classified as the most similar pattern is used instead of the scene images of the new items to which appropriate item position information is not given, it is expected that the same effect will be obtained in parameter calculation. The item position information is given to a part of the scene image of the new item whose recognition result using the representative parameter value of the most similar pattern is judged to be a failure in the process of step S30. Therefore, by using the item position information and the teaching data of the item classified as the most similar pattern as input data for parameter calculation, a parameter value that sufficiently increases the recognition performance of the new item can be obtained.

[0044] The second example of the simultaneous optimization is an example in which the range of search in the parameter space is limited to the vicinity of the representative parameter value of the most similar pattern. In this case, the input teaching data is a scene image of a new item whose recognition result using the representative parameter value of the most similar pattern is judged to be a failure. The scene image may be given item position information in the process of step S30. It is expected that the parameter values ​​in the vicinity of the representative parameter value of the most similar pattern can correctly recognize the scene image of a new item whose recognition result using the representative parameter value of the most similar pattern is judged to be a success, similar to the representative parameter value of the most similar pattern. Therefore, if a parameter value that can correctly recognize the input teaching data is found within the range of the scene image judged to be successful, it is a parameter value that sufficiently improves the recognition performance of the new item. In this way, the search range can be restricted by simultaneous optimization, and the computational costs can be reduced.

[0045] Next, the recognition performance evaluation unit 408 judges whether the calculated parameter values ​​are sufficient for recognizing the new article (S50). If the judgment is made while taking into consideration the recognition of a scene image of a new article to which no article position information is attached, for example, a method such as that shown in FIG. 7 can be used.

[0046] If it is determined in step S50 that the item is recognizable (S50: YES), the calculated parameter value is determined as a recognizable parameter value for the new item (S51). Even if the representative parameter value cannot be used as is, by utilizing the information of the pattern that is most useful for calculating the appropriate parameter value, it is possible to determine the appropriate parameter value even if the item position information manually added to the scene image of the new item is small. FIG. 11 shows an example of this process, in which pattern A is selected as the most similar pattern to the new item. As shown in FIG. 11, the representative parameter value of pattern A is not included in the parameter range that satisfies the required recognition performance of the new item, but is closer to the representative parameter value range of pattern A than the representative parameter value of pattern B. Then, the parameter value that allows the new item to be recognized by simultaneous optimization is found relatively close to the representative parameter value of pattern A. It is not necessary that the parameter value at this stage allows the item group classified into pattern A to be recognized.

[0047] On the other hand, if it is determined in step S50 that the object is not recognizable (S50: NO), the object position information is assigned to some of the scene images of the new object to which the object position information has not been assigned (S60). For example, the method shown in FIG. 9 can be used to achieve this.

[0048] Next, parameter calculation is performed (S70). Here, a model of the new object and a scene image of the new object to which object position information is added are sufficient as input data. At this point, there is more teaching data of the new object available than at the time of processing in step S40, and it is highly likely that parameter values ​​that sufficiently improve the recognition performance of the new object will be calculated. Here, this optimization is called individual optimization to distinguish it from simultaneous optimization.

[0049] Then, the recognition performance evaluation unit 408 judges whether the calculated parameter values ​​are sufficient for recognizing the new article (S80). If the judgment is made while taking into consideration the recognition of a scene image of a new article to which no article position information is attached, for example, a method as shown in FIG. 7 can be used.

[0050] If it is determined in step S80 that the new object is recognizable (S80: YES), the calculated parameter values ​​are determined as the recognizable parameter values ​​of the new object (S81). Figure 12 shows an example of this process, where pattern A is selected as the most similar pattern of the new object. However, since the performance of the parameter values ​​calculated in the joint optimization is not good, the parameter values ​​calculated in the individual optimization of the new object are determined as the recognizable parameter values ​​of the new object.

[0051] On the other hand, if it is determined in step S80 that the new object is not recognizable (S80: NO), this means that the currently assumed recognition method cannot achieve sufficient performance for the new object, and, for example, the new object is excluded from the parameter setting targets, and the series of processes is terminated.

[0052] As described above, the parameter determination process of this embodiment selects a pattern from a group of registered items that is expected to be most useful in determining appropriate parameter values ​​for a new item, and utilizes that information, making it possible to determine appropriate parameters even if the amount of teaching data for the new item is small.

[0053] <Hardware configuration> An example of a hardware configuration of the item recognition processing device will be described with reference to Fig. 13. The item recognition processing device 1 is a computer configured by connecting a processor 101, a memory 102, an auxiliary storage device 103, an input device 104, an output device 105, and a communication IF (Interface) 106 to an internal bus 107.

[0054] The processor 101 executes a program stored in the memory 102 to realize an appropriate parameter calculation process. In this embodiment, a program that executes various functions related to the parameter calculation process (see FIG. 4) is assumed, but various other programs may also be executed. Input data and output data for executing the program are mainly stored in the memory 102. The memory 102 may be composed of, for example, a non-volatile storage element (e.g., a ROM (Read Only Memory)) for storing a program that does not need to be changed, and a volatile storage element (e.g., a RAM (Random Access Memory)) for temporarily storing the program to be executed and data used when the program is executed. The auxiliary storage device 103 includes a non-volatile large-capacity storage device such as a magnetic storage device (HDD: Hard Disk Drive), and stores the program executed by the processor 101 and data (including DB, etc.) used when the program is executed. The program for the parameter calculation process according to this embodiment is stored in the auxiliary storage device 103 in advance, read out from there, loaded into the memory 102, and executed by the processor 101.

[0055] The input device 104 is an input unit such as a keyboard or a mouse that accepts input from an operator, and enables program execution and input data specification. The output device 105 is a device that visually outputs the results of program execution, such as a display device that displays information or a printer that prints information. In this embodiment, the display device displays information in a format that can be recognized by the operator, thereby realizing the processing shown in Fig. 8 and Fig. 10. The communication IF 106 is a network interface device that connects the recognition processing device 1 to other devices and controls communication.

[0056] This hardware configuration may be configured as a single computer, or one or more of the components, namely, the processor 101, memory 102, auxiliary storage device 103, input device 104, output device 105, and communication IF 106, may be configured as one or more computers connected via a network. EXAMPLES

[0057] In the second embodiment, a pattern is updated at the same time as the parameters of the parameter determination target article in the first embodiment. In the first embodiment, it is assumed that there is already available pattern classification information when a new article is registered, but in the second embodiment, a specific method for constructing the appropriate pattern classification information is presented.

[0058] FIG. 14 illustrates an example of a parameter determination process according to the second embodiment. Steps S500, S210, S510, and S810 are different from those in Fig. 5 (parameter determination processing procedure) according to the first embodiment. The processing in these steps enables appropriate pattern updating accompanying registration of a new item, in addition to parameter determination of the new item. In the following, the description of the same processing as in Fig. 5 will be omitted, and the pattern updating processing operation and its action and effect according to the characteristic processing steps in Fig. 14 will be described.

[0059] In step S210, not only are representative parameter values ​​of the most similar pattern selected as parameter values ​​with which the new item can be recognized, but also the pattern into which the new item is classified is set as the most similar pattern. An example of this process is shown in Figure 15. Since pattern A is selected as the most similar pattern to the new item, the representative parameter values ​​of pattern A are selected as parameter values ​​with which the new item can be recognized, and the new item is added to pattern A.

[0060] Next, in step S500, it is determined whether the parameter values ​​calculated by the simultaneous optimization in step S40 are capable of simultaneously recognizing a group of objects classified into the most similar pattern in addition to a new object. Here, when making the determination while taking into consideration the recognition of a scene image to which no object position information is attached, this can be realized, for example, by the method shown in FIG.

[0061] Step S510 is performed when it is determined that the parameter value calculated in step S500 can simultaneously recognize not only the new item but also the group of items classified into the most similar pattern. In addition to determining the parameter value as a recognizable parameter value for the new item, the new item is classified into the most similar pattern, and the representative parameter value of the most similar pattern is replaced with the calculated parameter value. An example of this process is shown in FIG. 16. Pattern A is selected as the most similar pattern for the new item, but it is determined that the representative parameter value is not good enough to recognize the new item, so simultaneous optimization is performed to calculate a new parameter value. The value is included in both the parameter range that satisfies the required recognition performance of the group of items in pattern A and the parameter range that satisfies the required recognition performance of the new item, and can be used as the new representative parameter value for pattern A. The new item is then classified into pattern A.

[0062] The determination in step S500 shows that this pattern update satisfies the criteria for pattern classification in the second embodiment. Furthermore, this pattern update can incorporate a new article without changing the number of patterns. This pattern update is preferable because an increase in the number of patterns increases the amount of processing in steps S20 and S500. Furthermore, the representative parameter values ​​can also be updated, allowing for flexible pattern updating. The representative parameter values ​​can be calculated by simultaneous optimization even when the amount of teaching data for the new article is small.

[0063] Finally, in step S810, not only are the parameter values ​​calculated in the individual optimization in step S70 determined as parameter values ​​capable of recognizing the new item, but also a new pattern is generated in which only the new item belongs to the pattern and the representative parameter values ​​are calculated. FIG. 17 shows an example of this process. Pattern A is selected as the most similar pattern to the new item, but the recognition of the new item by the representative parameter value is deemed to be poor, so simultaneous optimization is performed to calculate new parameter values. The parameter values ​​calculated in the simultaneous optimization are sufficient to recognize the new item. However, all items classified into pattern A cannot be sufficiently recognized, so individual optimization is performed in Example 2. The parameter values ​​calculated in the individual optimization are sufficient to recognize the new item, so the parameter values ​​are determined as parameter values ​​capable of recognizing the new item. Furthermore, a new pattern C is generated, the new item is classified into it, and the representative parameter values ​​are those calculated in the individual optimization.

[0064] By updating the patterns in step S810, new objects that are determined not to be classified into any patterns are classified into the newly generated pattern. In particular, it is possible to construct a pattern even when there is no registered object.

[0065] Here, the function for performing the above pattern update process will be described. The functions different from those in the first embodiment will be described with reference to FIG. 4. The recognition performance evaluation unit 408 evaluates the recognition rate for the new item of the parameter values ​​calculated by the optimization unit, and checks whether it satisfies the required value. In the case of simultaneous optimization, the recognition rates are evaluated not only for the new item, but also for multiple items (item groups) belonging to the most similar pattern. If the result of simultaneous optimization does not satisfy the required value, the process returns to the annotation signal receiving unit 406.

[0066] The pattern information DB 410 stores information on registered object groups (model information and scene images) and pattern information extracted from them. In the second embodiment, the information is appropriately changed when outputting optimal parameters for a new object.

[0067] As described above, according to the parameter determination and update of the second embodiment, by selecting a pattern from the registered article group that is expected to be most useful in determining appropriate parameter values ​​for a new article, and utilizing that information, it is possible to determine appropriate parameters for the new article even if the amount of teaching data for the new article is small. Furthermore, it is possible to update the pattern so that it becomes a useful pattern for the article for which parameters are to be determined later. EXAMPLES

[0068] Example 3 presents parameter determination that can repeat simultaneous optimization processing in parameter determination of Example 1. Figure 18 shows an example of a parameter determination processing procedure according to Example 3. In comparison with Example 1, step S52 is newly added to the processing procedure of Figure 4, which enables loop processing. Steps other than step S52 and the functions that realize them are substantially the same as those of Example 1, so that their description will be omitted.

[0069] In this embodiment, if it is determined in step S50 that the parameter values ​​calculated by the simultaneous optimization in step S40 are not sufficient to recognize the new article (S50: NO), it is determined whether article position information has been added to all of the scene images of the new article that were determined to have failed in recognition using the representative parameter values ​​of the most similar pattern in step S20 (S52).If the result of this determination is that article position information has been added to all of them (S52: YES), the process proceeds to step S60, and thereafter the same processing procedure as in embodiment 1 is followed.

[0070] On the other hand, if the result of the above judgment is that there is any scene image to which no item position information has been assigned (S52: NO), the process returns to step S30, and item position information can be assigned to the scene image to which no item position information has been assigned. That is, simultaneous optimization is performed in step S40, but the difference from the previous simultaneous optimization is that the number of scene images to which item position information of a new item has been assigned has increased this time. Therefore, the calculated parameter value is more likely to be able to sufficiently recognize the new item in step S50. If the new item cannot be sufficiently recognized, and there are still scene images to which item position information can be assigned, the process returns to step S30 again in the process of step S52, and the above-mentioned process can be repeated. The possibility of exiting the repeated process is when it is determined that the parameter value capable of sufficiently recognizing the new item has been calculated in step S50 (step S51), and it is determined that item position information has been assigned to all failure scene images, or when it is determined that there is no need to perform further simultaneous optimization in step S51.

[0071] Thus, according to the third embodiment, compared to the first embodiment, it is possible to gradually assign item position information to new items until the results of the simultaneous optimization become favorable, thereby achieving the advantage of being able to obtain parameter values ​​capable of adequately recognizing new items with as little cost as possible in creating teaching data. EXAMPLES

[0072] The fourth embodiment is an example in which a plurality of representative parameters (called a representative parameter value group) is considered as the representative parameter value.

[0073] [Response to Example 1] In the recognition process in the first embodiment (see FIG. 5), each pattern has one representative parameter value. However, it is possible to find multiple parameter values ​​that make the recognition performance of the belonging item equal to or higher than the required performance, and in this embodiment, it is possible to assign multiple representative parameter values ​​to each pattern.

[0074] For example, in selecting the most similar pattern, the recognition performance of the new article is evaluated for all representative parameter values ​​for each pattern, and the pattern to which the representative parameter value with the best recognition performance belongs is selected as the most similar pattern. Although the calculation cost and the cost required for determining the recognition result increase as the number of parameter value evaluations increases, more granular determination is possible. In this case, the representative parameter value used in determining the optimal parameter value for the new article can be the representative parameter value with the highest recognition rate that was focused on when selecting the most similar pattern among the multiple representative parameter values ​​of the most similar patterns.

[0075] [Response to Example 2] In the above embodiment 1, an example was described in which a plurality of representative parameter values ​​for each pattern are considered in order to realize the selection of the most similar pattern with finer granularity and the determination of the optimal parameter value with higher recognition performance. A similar example can be described in the embodiment 2. In this case, in addition to the changes in the embodiment 1, a change is required to the process related to the update of pattern information. For example, when changing the representative parameter value of the most similar pattern in step S510, a plurality of representative parameter values ​​are changed, and when adding a new pattern in step S810, a plurality of representative parameter values ​​are obtained. For the former, steps S40 and S500 are repeated to obtain the necessary number of representative parameter values, and the necessary number of representative parameter values ​​are replaced with the existing representative parameter values ​​of the most similar pattern. Similarly, for the latter, steps S70 and S80 are repeated until the necessary number of representative parameter values ​​are obtained. Here, if at least one new representative parameter value has been obtained, it is expected that other values ​​that can become the representative parameter value are near that value, so it is possible to limit the search range to the vicinity of that value and suppress the increase in the calculation cost to some extent. EXAMPLES

[0076] With reference to Fig. 19, an example in which the article recognition processing device according to the above-mentioned Examples 1-4 is applied to article recognition by a picking robot will be described. In Fig. 19, a computer realizing the article recognition processing device 1 has a mouse as an input device 104 and a display as an output device 105. A robot 1902 and an image sensor 1903 are connected to a communication IF 106 of the article recognition processing device 1 via a communication line 1901. The robot 1902 receives the recognition result by the article recognition processing device 1 and performs a desired operation. The image sensor 1903 captures an object 1931 that is the target of the operation of the robot 1902 and a work scene for the object. The sensor and the computer may be part of the robot.

[0077] The flow of operations of the robot shown in FIG. 19 is as follows, for example. First, the item recognition processing device 1 executes recognition processing on an image acquired by a sensor 1903 installed on the main body of the robot 1902 or in its vicinity, and obtains a recognition result. Then, the computer analyzes the recognition result with an appropriate program, and obtains information for inducing an appropriate robot operation. Based on the information, the robot 1902 is made to perform an operation. For example, the object 1931 to be recognized is grasped, and the grasped object is moved to a predetermined position. In addition, although the work scene in FIG. 19 is depicted in a fixed state, the work scene can be changed as appropriate. In particular, it is possible to use a situation that changes from moment to moment, such as an actual production line.

[0078] When the first embodiment is applied to this application example, the following effect is obtained. That is, the success or failure of the recognition result, which occurs in step S10 or the like, can be determined based on the movement of the robot 1902. As shown in FIG. 7, when the recognition result is displayed on a display device, it cannot be denied that the operator may determine that the recognition is successful even if the accuracy is insufficient to achieve the purpose, or that the recognition is unsuccessful even if the accuracy is sufficient. Therefore, according to this application example, a more appropriate judgment can be made by making the robot perform the actual intended movement. [Explanation of symbols]

[0079] 1: Item recognition processing device 101: Processor 102: Memory 103: Auxiliary storage device 104: Input device 105: Output device 106: Communication IF 107: Internal bus 401: Reception unit 402: Reception unit for signal of success or failure of recognition 403: Representative parameter value evaluation unit 404: Most similar pattern determination unit 405: Pattern information DB 406: Annotation signal reception unit 407: Optimization unit 408: Recognition performance evaluation unit 409: Output section 410: Storage section

Claims

1. A recognition processing method in which a computer executes an article recognition process, storing a plurality of patterns including a plurality of registered articles, each pattern having a representative parameter value, in a storage unit; receiving new item information having at least a model of the new item and a plurality of scene images; a recognition processing step of performing a recognition process on a plurality of scene images of the novel article using the representative parameter values ​​of each of the plurality of patterns; selecting the pattern corresponding to the representative parameter value determined to be the best as a result of the recognition process as the most similar pattern to the new article; If a result of a recognition process for the new article using the representative parameter value of the selected most similar pattern is good, determining the representative parameter value as a recognizable parameter value of the new article; The recognition processing method includes:

2. If the result of the recognition process (re-recognition process) of the new article using the representative parameter value of the most similar pattern is not good, selecting at least one of the plurality of scene images of the novel article that is determined to have failed as a result of the recognition process; Adding item position information to the selected scene image; calculating new parameter values ​​based on the selected scene image, the item position information, the model of the new item, and the representative parameter values ​​of the most similar pattern or information for calculating the representative parameter values ​​of the most similar pattern; performing a recognition process for the new article on the plurality of scene images of the new article using the model of the new article and the new parameter values; If the result of the recognition process is good, determining the new parameter value as the recognizable parameter value of the new article; The recognition processing method according to claim 1

3. Each of the plurality of patterns has, as pattern information, at least models of all the registered articles classified into the pattern and a plurality of scene images in which all the registered articles classified into the pattern are captured; if a result of the recognition process using the representative parameter value of the most similar pattern is good, the model of the new article and the plurality of scene images in which the new article is captured are added to the pattern information of the most similar pattern, and the new article is classified into the most similar pattern; if the result of the recognition process (re-recognition process) of the new item is good for the plurality of scene images of the new item using the model of the new item and the new parameter values, for each of all registered items classified into the most similar pattern, a recognition process of the registered item is performed for the plurality of scene images in which the registered item is photographed using the model of the registered item and the new parameter values; if the results of the recognition process for all of the registered articles classified into the most similar pattern are good, add the model of the new article and the plurality of scene images in which the new article is photographed to the pattern information of the most similar pattern; replacing the representative parameter value of the most similar pattern with the new parameter value, and classifying the new article into the most similar pattern; The recognition processing method according to claim 2 .

4. If the results of all of the recognition processes are not good, selecting a portion of the plurality of scene images of the novel item; Adding item position information to each of the selected scene images; Calculating new parameter values ​​based on a portion of the plurality of scene images of the new item to which item position information is assigned, the item position information, and the model of the new item; performing a recognition process (re-recognition process) of the new article on the plurality of scene images of the new article using the model of the new article and the new parameter values; If the result of the re-recognition process is good, the new parameter values ​​are determined as recognizable parameter values ​​of the new article; retaining at least the model of the new article and the plurality of scene images in which the new article is captured as pattern information, creating a new pattern in which the new parameter value is a representative parameter value, and classifying the new article into the new pattern; The recognition processing method according to claim 3.

5. displaying, on a display device, information on the estimated position of the article relative to the scene image of the article together with the scene image; inputting a result of the recognition processing based on the display through an input device; The recognition processing method according to claim 1 .

6. The target is to determine the parameters of the robot's object recognition process. the robot executes a predetermined operation based on estimated position information of the object estimated by object recognition for a scene image of the object, the object recognition being executed using the recognition process, a model of the object, and arbitrary parameter values; determining a result of the recognition process based on the motion; The recognition processing method according to claim 1 .

7. When the result of the re-recognition process is not good and the result of the recognition process using the representative parameter value of the most similar pattern is determined to be a failure, there is a scene image of the new article to which no article position information is assigned, and the process according to claim 2 is executed again. The recognition processing method according to claim 2 .

8. each of the plurality of patterns is a representative parameter value group having a plurality of parameter values ​​that enable recognition of a plurality of the registered articles classified into the corresponding pattern; performing a recognition process for the new article on the plurality of scene images of the new article by using the representative parameter value group of each of the plurality of patterns; selecting the pattern corresponding to the representative parameter value group having the best parameter value as a result of the recognition process as the most similar pattern to the new article; determining, when a result of the recognition process using the parameter values ​​belonging to the representative parameter value group of the most similar pattern is good, the parameter values ​​belonging to the representative parameter value group of the most similar pattern as recognizable parameter values ​​of the novel article; The recognition processing method according to claim 1 .

9. A storage unit that stores a plurality of patterns including a plurality of registered articles, each having a representative parameter value; A receiving unit that receives at least a model of a new item and information on the new item having a plurality of scene images; a recognition processing unit that performs a recognition process of a plurality of scene images of the novel article by using the representative parameter values ​​of each of the plurality of patterns; a determination unit that selects the pattern corresponding to the representative parameter value that is determined to be the best as a result of the recognition processing by the recognition processing unit as the most similar pattern of the new article; an output unit that outputs the representative parameter value as a recognizable parameter value of the new article when a result of a recognition process of the new article using the representative parameter value of the selected most similar pattern is good; A recognition processing device having the above configuration.

10. If the result of the recognition process of the new article using the representative parameter value of the most similar pattern is not good, selecting at least one of the plurality of scene images of the novel article that is determined to have failed as a result of the recognition process; Adding item position information to the selected scene image; the determination unit calculates new parameter values ​​based on the selected scene image, the item position information, the model of the new item, and the representative parameter values ​​of the most similar pattern or information for calculating the representative parameter values ​​of the most similar pattern; the recognition processing unit performs a recognition process of the new article on the plurality of scene images of the new article by using the model of the new article and the new parameter values; When a result of the recognition process is good, the determination unit determines the new parameter value as a recognizable parameter value of the new article. The recognition processing device according to claim 9 .

11. Each of the plurality of patterns has, as pattern information, at least models of all the registered articles classified into the pattern and a plurality of scene images in which all the registered articles classified into the pattern are captured; when a result of the recognition process using the representative parameter value of the most similar pattern is good, the determination unit adds the model of the new article and the plurality of scene images in which the new article is captured to the pattern information of the most similar pattern, and classifies the new article into the most similar pattern; the recognition processing unit, when a result of a recognition process (re-recognition process) of the new item for the plurality of scene images of the new item using the model of the new item and the new parameter values ​​is good, performs a recognition process of the registered item for each of all registered items classified into the most similar pattern, using the model of the registered item and the new parameter values ​​for the plurality of scene images in which the registered item is photographed; when a result of the recognition process of all the registered articles classified into the most similar pattern is good, the determination unit adds the model of the new article and the plurality of scene images in which the new article is photographed to the pattern information of the most similar pattern; replacing the representative parameter value of the most similar pattern with the new parameter value, and classifying the new article into the most similar pattern; The recognition processing device according to claim 10.

12. If the results of all of the recognition processes are not good, selecting a portion of the plurality of scene images of the novel item; Adding item position information to each of the selected scene images; The determination unit calculates new parameter values ​​based on a part of the plurality of scene images of the new item to which item position information is assigned, the item position information, and the model of the new item; the recognition processing unit executes a recognition process (re-recognition process) of the new article on the plurality of scene images of the new article using the model of the new article and the new parameter values; The determination unit determines the new parameter values ​​as recognizable parameter values ​​of the new article when a result of the re-recognition process is good, retaining at least the model of the new article and the plurality of scene images in which the new article is captured as pattern information, creating a new pattern in which the new parameter value is a representative parameter value, and classifying the new article into the new pattern; The recognition processing device according to claim 11.

Images