Parameter determination method, data recognition device, and parameter determination device

Abstract

It assists in the recognition processing of data that is unsuitable for the learning models built into detectors and classifiers. [Solution] In a parameter determination method used in a rule classifier that performs rule-based identification processing based on determination conditions on data regions to identify a predetermined object, the method detects data regions that are likely to be objects from the input data and obtains scores from a learning model that generates scores indicating the likelihood of the data regions being objects. Multiple feature quantities are calculated for data regions that generate scores outside a predetermined range. A combination of feature quantity types that makes it easy to separate the multiple data regions that generate scores outside the predetermined range into a first data region group consisting of data regions of objects and a second data region group consisting of data regions that are not objects is selected, and the parameters are determined based on the selected combination of feature quantity types.

Description

Technical Field

[0001] The present invention relates to a parameter determination method, a data recognition device, and a parameter determination device.

Background Art

[0002] The following Patent Document 1 describes a technique for detecting an object appearing in a captured image by image processing using a learning model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Although the performance of detectors and classifiers has improved dramatically through data recognition processing using learning models, in scenarios where erroneous determinations by detectors and classifiers are not permitted, it is necessary to appropriately control (adjust) the detectors and classifiers to prevent erroneous determinations. However, controlling the learning model itself (for example, re-learning the learning model) is not easy and is not realistic from the perspective of cost and labor. The present invention has been made in view of the above problems, and an object thereof is to assist recognition processing for data that is not suitable for the recognition processing of a learning model provided in a detector or a classifier.

Means for Solving the Problems

[0005] According to a first embodiment of the present invention, a parameter determination method is provided for determining parameters that define a determination condition, which is used in a rule classifier that performs rule-based identification processing based on a determination condition on a data region and identifies a predetermined target for detection. The parameter determination method includes a score acquisition step of obtaining a score from a learning model that detects data regions that are likely to be targets for detection from input data and generates a score indicating the likelihood of the data regions being targets for detection; a feature calculation step of calculating multiple features for data regions for which a score outside a predetermined range has been generated; a feature selection step of selecting a combination of feature types that makes it easy to separate the multiple data regions for which a score outside a predetermined range has been generated into a first data region group consisting of data regions to be detected and a second data region group consisting of data regions that are not to be detected; and a parameter determination step of determining parameters based on the selected combination of feature types.

[0006] According to a second embodiment of the present invention, a data recognition device is provided, which includes a learning model that detects data regions that are likely to be a predetermined target from input data and generates a score indicating the likelihood of the data region being a target, the learning model being trained using learning data and ground truth data of the location of a target on the learning data given to the learning data, and a rule classifier that performs rule-based identification processing based on judgment conditions determined by parameters determined by the parameter determination method of the first embodiment for data regions from which a score outside a predetermined range has been generated among the detection results by the learning model.

[0007] According to a third embodiment of the present invention, a parameter determination method is provided for determining parameters that define a determination condition, which is used in a rule classifier that performs rule-based identification processing based on a determination condition on input data and identifies the class of the input data. The parameter determination method includes a score acquisition step of obtaining a score from a learning model that generates a score indicating the likelihood of the input data belonging to a predetermined class; a feature calculation step of calculating a plurality of features for input data for which a score outside a predetermined range has been generated; a feature selection step of selecting a combination of feature types that makes it easy to separate the plurality of input data for which a score outside a predetermined range has been generated into a first data group consisting of input data of a predetermined class and a second data group consisting of input data other than the predetermined class; and a parameter determination step of determining parameters based on the selected combination of feature types.

[0008] According to a fourth embodiment of the present invention, a data recognition device is provided, comprising: a learning model that generates a score indicating the likelihood of input data belonging to a predetermined class, the learning model being trained using training data and ground truth data of the class of the training data given to the training data; and a rule classifier that performs rule-based identification processing based on judgment conditions determined by parameters determined by the parameter determination method of the third embodiment for input data from which the learning model has generated a score outside a predetermined range. [Effects of the Invention]

[0009] According to the present invention, it is possible to assist in the recognition processing of data that is unsuitable for the recognition processing of the learning model provided in the detector or classifier. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing a schematic configuration example of an image recognition device according to an embodiment. [Figure 2] This figure shows an example of an input image that is processed by a learned model. [Figure 3](a) is a figure showing an example of a high-scoring correct image region, (b) is a figure showing an example of a low-scoring correct image region, (c) is a figure showing an example of a low-scoring incorrect image region, and (d) is a figure showing an example of a high-scoring incorrect image region. [Figure 4] This is an explanatory diagram of the scores for candidate rectangular image regions. [Figure 5] This is a block diagram of an example of the functional configuration of an image recognition device according to the first embodiment. [Figure 6] This is a block diagram showing a schematic configuration example of a parameter determination device according to an embodiment. [Figure 7] This is a block diagram of an example of the functional configuration of the parameter determination device according to the embodiment. [Figure 8] (a) and (b) are schematic diagrams of examples of the segment area and keypoint coordinates of an object within a candidate rectangular region. [Figure 9] This is a schematic diagram showing the distribution of low-scoring image regions in the feature space. [Figure 10] This is a flowchart of an example of a parameter determination method in the first embodiment. [Figure 11] This is a block diagram of an example of the functional configuration of an image recognition device in the first modified form. [Figure 12] This is a block diagram of a functional configuration example of an image recognition device in the second modified form. [Figure 13] (a) to (c) are explanatory diagrams illustrating an example of a parameter determination method for the third modified example. [Figure 14] This is a block diagram of an example of the functional configuration of an image recognition device in the third modified example. [Figure 15] This is a block diagram of an example of the functional configuration of an image recognition device according to the second embodiment. [Figure 16] This is a block diagram of an example of the functional configuration of the parameter determination device according to the second embodiment. [Figure 17] This is a flowchart of an example of a parameter determination method in the second embodiment. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings. The embodiments of the present invention described below are illustrative examples of devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention is not limited to the structure, arrangement, etc., of the components described below. Various modifications can be made to the technical concept of the present invention within the technical scope defined by the claims described in the patent claims.

[0012] (First Embodiment) Figure 1 is a block diagram showing a schematic configuration example of an image recognition device according to the embodiment. The image recognition device 1 of the first embodiment detects (identifies) a predetermined object in the input image. The following describes an example in which a person is detected as the object, but the object detected by the image recognition device 1 is not limited to a person, but may be a vehicle, an animal other than a person, a suspicious object, etc. The input image is an example of the "input data" described in the claims, and the image recognition device is an example of the "data recognition device" described in the claims. The image recognition device 1 consists of an imaging unit 2, a communication unit 3, a storage unit 4, an image processing unit 5, a display unit 6, and an operation input unit 7.

[0013] The imaging unit 2 is a camera that acquires images, and in this embodiment, it is a surveillance camera. The imaging unit 2 is connected to the image processing unit 5 via the communication unit 3, and generates images by photographing the surveillance space at predetermined time intervals, and sequentially inputs the generated images to the image processing unit 5. For example, the imaging unit 2 is installed in a corner of a room that is the surveillance space, with a predetermined fixed field of view that provides an overview of the surveillance space, and generates color images by photographing the surveillance space with a frame period of 1 second. The imaging unit 2 may also be a mobile camera attached to a robot or a person's head. In addition, monochrome images or thermal images may be generated instead of color images. Furthermore, three-dimensional or more image data such as depth images may also be used.

[0014] The communication unit 3 is a communication circuit, with one end connected to the image processing unit 5 and the other end connected to the imaging unit 2, the display unit 6, and the operation input unit 7. The communication unit 3 acquires an image from the imaging unit 2 and inputs it to the image processing unit 5 as an input image Im. It also acquires user instructions and other information from the operation input unit 7 and inputs them to the image processing unit 5. Furthermore, the communication unit 3 receives the object detection results from the image processing unit 5 and outputs them to the display unit 6.

[0015] The imaging unit 2, communication unit 3, storage unit 4, image processing unit 5, display unit 6, and operation input unit 7 are connected to each other in a manner appropriate to the installation location of each unit. For example, if the imaging unit 2, communication unit 3, and image processing unit 5 are installed remotely, the imaging unit 2 and communication unit 3 can be connected via an internet line. Alternatively, the communication unit 3 and image processing unit 5 can be connected via a bus. Other connection methods include LAN (Local Area Network) and various cables.

[0016] The memory unit 4 is a memory device such as ROM (Read Only Memory) or RAM (Random Access Memory) that stores various programs and data. For example, the memory unit 4 also stores information about a learning model that performs an object detection task and inputs and outputs this information to and from the image processing unit 5. In other words, information necessary for object detection processing and information generated during the process of said processing are input and output between the memory unit 4 and the image processing unit 5.

[0017] The image processing unit 5 is composed of computing devices such as a CPU (Central Processing Unit), DSP (Digital Signal Processor), MCU (Micro Control Unit), and GPU (Graphics Processing Unit). The image processing unit 5 operates as various processing and control means by reading and executing programs from the storage unit 4, and reads various data from the storage unit 4 as needed and stores the generated data in the storage unit 4. For example, the image processing unit 5 learns and generates a learning model and stores the generated learning model in the storage unit 4 via the communication unit 3. The image processing unit 5 also uses the learning model read from the storage unit 4 to detect objects.

[0018] The display unit 6 is a liquid crystal display or an organic EL (Electro-Luminescence) display, etc., and displays the object detection results input from the image processing unit 5 via the communication unit 3. The operation input unit 7 is an input device for the image processing unit 5 and consists of a keyboard, mouse, and the like.

[0019] The image recognition device 1 includes a learning model that detects image regions in the input image Im in which a candidate for a predetermined object (person) is depicted, and generates a score indicating the likelihood (plausibility) that the subject in the detected image region is the object. In other words, the learning model detects image regions that are likely to be objects from the input image Im and generates a score indicating the likelihood that the image region is an object. The object is an example of the "detection target" described in the claims, and the image region that is likely to be an object is an example of the "data region" described in the claims. Figure 2 shows an example of an input image Im that is processed by the learning model of the image recognition device 1.

[0020] The learning model of the image recognition device 1 detects candidate objects (people) to be detected from the input image Im and outputs bounding boxes B1 to B5 that surround the objects. In the following description, the bounding boxes B1 to B5 that surround the objects detected from the input image Im will be referred to as "candidate rectangles," and the area of ​​the input image Im enclosed by the candidate rectangles will be referred to as the "candidate rectangle image region." The candidate rectangle image region is an example of the "image region" described in the claims.

[0021] Figures 3(a) to 3(c) show the candidate rectangular image regions enclosed by rectangles B1, B3, and B5, respectively, which were detected from the input image Im by the learning model. The candidate rectangular image region IcH shown in Figure 3(a) is an image region that contains an object (person) and has a relatively high score. In the following explanation, the candidate rectangular image region IcH will be referred to as the "high-scoring correct image region IcH". In the following explanation, the image region containing an object may be referred to as the "image region of the object".

[0022] The candidate rectangular image region IcL shown in Figure 3(b) is an image region of the target object that has a relatively low score. In the following explanation, the candidate rectangular image region IcL will be referred to as the "low-score ground truth image region IcL". The candidate rectangular image region IwL shown in Figure 3(c) is an image region that is not the target object and has a relatively low score.

[0023] Image regions that do not represent the target object include not only image regions that contain objects other than the target object, but also regions that are inappropriate as to represent the target object because the target object is not fully captured, such as the candidate rectangular image region enclosed by rectangle B2. In the following explanation, the candidate rectangular image region IwL will be referred to as the "low-score incorrect image region IwL".

[0024] Thus, candidate rectangular image regions with low scores (i.e., image regions with a low likelihood of containing the object) may include not only low-scoring incorrect image regions IwL, which are image regions that do not contain the object, but also low-scoring correct image regions IcL, which contain the object. For this reason, it is difficult for the learning model to distinguish between these and extract the low-scoring correct image regions IcL as the detection target.

[0025] Figure 4 is an explanatory diagram of the scores of candidate rectangular image regions. Candidate rectangular image regions detected by the learning model have a score of Th1 or higher than the rectangle detection threshold. Low-scoring correct image region IcL and low-scoring incorrect image region IwL have scores greater than or equal to the rectangle detection threshold Th1 and less than the object detection threshold Th2.

[0026] High-scoring ground truth image regions IcH have a score equal to or greater than the object detection threshold Th2. In the following explanation, candidate rectangular image regions with a score equal to or greater than the rectangle detection threshold Th1 but less than the object detection threshold Th2 will be referred to as "low-scoring image regions," and candidate rectangular image regions with a score equal to or greater than the object detection threshold Th2 will be referred to as "high-scoring image regions."

[0027] If only objects that appear in candidate rectangular image regions with a score of Th2 or higher are detected as objects, there is a risk of missing detections (missed reports) in low-scoring ground truth image regions IcL. Therefore, the image recognition device 1 of this embodiment includes a rule classifier that identifies the low-scoring correct image region IcL from the low-scoring image region by rule-based identification processing based on determination conditions.

[0028] A score below the object detection threshold Th2 is an example of a "score outside the specified range" as described in the claims. In this way, by using a rule classifier to detect low-scoring ground truth image regions (IcL) where the learned model has difficulty distinguishing whether or not it is an image region of the target object, it is possible to assist in recognition processing of images that are unsuitable for the learned model's image processing.

[0029] Figure 5 is a block diagram of an example of the functional configuration of the image recognition device 1 according to the first embodiment. The image recognition device 1 comprises an image input unit 10, an object detector 11, a score determination unit 12, a rule classifier 13a, and a detection result output unit 14. In Figure 1, the image processing unit 5 functions as an object detector 11, a score determination unit 12, and a rule classifier 13a. Furthermore, the communication unit 3 works in cooperation with the image processing unit 5 to function as an image input unit 10 and a detection result output unit 14.

[0030] The image input unit 10 sequentially acquires the images captured by the imaging unit 2 and inputs them to the object detector 11 as input images Im. The object detector 11 has a learning model that detects candidate rectangular image regions in the input image Im in which a predetermined object candidate is captured, and generates a score indicating the likelihood that the subject in the detected candidate rectangular image region is the predetermined object. For example, the learning model may be artificial intelligence (AI) realized by a neural network or the like, and may be a learning model trained using training image data and ground truth data of the object's position on the training image data given to the training image data. The object detector 11 outputs the detected candidate rectangular image region and its score to the score determination unit 12. The training image data is an example of the "training data" described in the claims.

[0031] The score determination unit 12 determines candidate rectangular image regions with a score equal to or greater than the object detection threshold Th2 as high-score image regions, and candidate rectangular image regions with a score less than the object detection threshold Th2 as low-score image regions, based on the score output by the object detector 11. The score determination unit 12 outputs the high-score image regions to the detection result output unit 14 and outputs the low-score image regions to the rule classifier 13a.

[0032] The rule classifier 13a identifies a predetermined object from an image region by performing rule-based identification processing based on judgment conditions on the input image region. Specifically, by performing rule-based identification processing on the low-scoring image region input from the score judgment unit 12, it determines whether the input low-scoring image region is a low-scoring correct image region IcL or a low-scoring incorrect image region IwL, and outputs the judgment result to the detection result output unit 14.

[0033] The decision criteria used in the rule-based classification process of the rule classifier 13a are defined by parameters relating to the image features of the input candidate rectangular image regions. In the following description, the parameters that define the rule-based decision conditions of the rule classifier 13a will be referred to as "rule parameters," and the image features related to the rule parameters will be referred to as "rule features." For example, the rule parameters may include the type of rule feature, its value, and a threshold th for determining whether the value of the rule feature satisfies the decision condition.

[0034] For example, the determination condition may be defined by rule parameters relating to multiple types of rule features. For example, the rule classifier 13a may determine whether the determination condition is met based on the weighted sum of the comparison results between the values ​​of the features included in the combination of multiple types of rule features and the threshold.

[0035] For example, when the candidate rectangular image region input to the rule classifier 13a is denoted as x, the multiple types of rule features F of the candidate rectangular image region x i If the value of the classifier H(x) in the following equation (1), calculated based on this, is "1", it can be determined that the candidate rectangular image region x is a low-score ground truth image region IcL, and if it is "-1", it can be determined that the candidate rectangular image region x is not a low-score ground truth image region IcL.

[0036]

number

[0037] In the above equation (1), h i (x) is a plurality of weak classifiers that identify whether or not the candidate rectangular image region x is a low-score correct image region IcL based on the rule feature amount of the candidate rectangular image region x, and the rule feature amount F i and the threshold th i based on the comparison result, it is identified whether the candidate rectangular image region x is the low-score correct image region IcL or not. The weak classifier h i (x) has a value of "1" when it is identified as the low-score correct image region IcL, and has a value of "-1" in other cases.

[0038] α i is the weighting coefficient of the weak classifier h i (x). sign(z) is a sign function that has a value of "1" when the variable z is greater than 0, and has a value of "-1" when the variable z is 0 or less. For example, the rule parameter may include the rule feature amount F i type and value, the threshold th i , and the weighting coefficient α i .

[0039] The detection result output unit 14 outputs the high-score image region as the detection result of the target object. Further, the detection result output unit 14 determines, based on the determination result of the rule classifier 13a, whether to output the low-score image region as the detection result of the target object or as a detection result other than the target object. For example, the detection result output unit 14 may output, as the detection result of the target object, the low-score image region determined by the rule classifier 13a as the low-score correct image region IcL, and output, as a detection result other than the target object, the low-score image region determined not to be the low-score correct image region IcL.

[0040] Further, for example, the detection result output unit 14 may correct the score of the low-score image region based on the determination result of the rule classifier 13a, output it as the detection result of the target object when the corrected score is equal to or greater than a predetermined threshold (for example, the target object detection threshold Th2), and output it as a detection result other than the target object when the corrected score is less than the predetermined threshold.

[0041] For example, the detection result output unit 14 may increase the score of the low-score ground truth image region IcL by adding an offset to the score of the low-score ground truth image region IcL or by multiplying it by a correction coefficient of 1 or more. The offset and correction coefficient may be fixed values, or they may be determined according to the following equation (2).

[0042]

number

[0043] Next, the method for determining the rule parameters in the first embodiment will be described. Figure 6 is a block diagram showing a schematic configuration example of the parameter determination device of the embodiment. The parameter determination device 20 consists of a communication unit 21, a storage unit 22, an image processing unit 23, a display unit 24, and an operation input unit 25.

[0044] The communication unit 21 is a communication circuit, with one end connected to the image processing unit 23 and the other end connected to the display unit 24 and the operation input unit 25. The communication unit 21 receives user instructions and other information from the operation input unit 25 and inputs them to the image processing unit 23. The communication unit 21 also receives rule parameters and their candidates determined by the image processing unit 23 and outputs them to the display unit 24.

[0045] The communication unit 21, storage unit 22, image processing unit 23, display unit 24, and operation input unit 25 are connected to each other in a manner appropriate to the installation location of each unit. For example, the communication unit 21 and the image processing unit 23 can be connected by a bus. Other connection means such as LAN (Local Area Network) and various cables can be used.

[0046] The memory unit 22 is a memory device such as ROM (Read Only Memory) or RAM (Random Access Memory) that stores various programs and data. For example, the memory unit 22 also stores an input image Im for extracting candidate rectangular image regions used to calculate rule features and for generating scores for the candidate rectangular image regions, and inputs and outputs this information to and from the image processing unit 23.

[0047] In other words, information used to determine the rule parameters and information generated during the processing are input and output between the storage unit 22 and the image processing unit 23. The input image Im stored in the memory unit 22 is an annotated image, and the positional information of a predetermined object depicted on the input image Im is provided in advance by the operator.

[0048] The image processing unit 23 is composed of computing devices such as a CPU (Central Processing Unit), DSP (Digital Signal Processor), MCU (Micro Control Unit), and GPU (Graphics Processing Unit). The image processing unit 23 operates as various processing and control means by reading and executing programs from the storage unit 22, and reads various data from the storage unit 22 as needed and stores the generated data in the storage unit 22.

[0049] The display unit 24 is a liquid crystal display or an organic EL (Electro-Luminescence) display, etc., and displays rule parameters and their candidates input from the image processing unit 23 via the communication unit 21. The operation input unit 25 is an input device for the image processing unit 23 and consists of a keyboard, mouse, and the like.

[0050] The parameter determination device 20 may be configured with the same hardware as the image recognition device 1 shown in Figure 1. In this case, the communication unit 3, storage unit 4, image processing unit 5, display unit 6, and operation input unit 7 shown in Figure 1 will operate as the communication unit 21, storage unit 22, image processing unit 23, display unit 24, and operation input unit 25 of the parameter determination device 20, respectively.

[0051] Figure 7 is a block diagram of an example of the functional configuration of the parameter determination device 20 of the first embodiment. The parameter determination device 20 comprises an image input unit 30, an object detector 31, a feature quantity calculation unit 32, a feature quantity selection unit 33, a parameter determination unit 34, and a manual adjustment unit 35. The image processing unit 23 in Figure 6 functions as an image input unit 30, an object detector 31, a feature calculation unit 32, a feature selection unit 33, and a parameter determination unit 34. In addition, the display unit 24 and the operation input unit 25 cooperate with the image processing unit 23 to function as a manual adjustment unit 35.

[0052] The image input unit 30 acquires the annotated input image Im stored in the storage unit 22 and inputs it to the object detector 31 and the feature calculation unit 32. The object detector 31 detects candidate rectangular image regions in the input image Im in which a predetermined candidate object is captured, and calculates a score for each candidate rectangular image region. In other words, the object detector 31 calculates a score for each candidate rectangular image region. The object detector 31 is an example of the "score acquisition unit" described in the claims.

[0053] The object detector 31 has a learning model that detects candidate rectangular image regions and generates a score for them. For example, the object detector 31 may have the same learning model as the object detector 11 shown in Figure 5. The object detector 31 outputs the detected candidate rectangular image region to the feature calculation unit 32, and outputs the score calculated for each candidate rectangular image region to the feature selection unit 33.

[0054] The feature calculation unit 32 calculates candidate features for multiple types of rule features. In the following description, the features calculated by the feature calculation unit 32 as candidate rule features may be referred to as "candidate features." For example, the feature calculation unit 32 may calculate the image features of the entire input image Im, that is, the global features of the entire input image Im, as candidate features. For example, the feature calculation unit 32 may calculate multiple types of image features as the image features of the entire input image Im, such as the brightness of the entire input image Im or the number of edges of the entire input image Im.

[0055] For example, the feature calculation unit 32 may calculate multiple types of candidate features for each candidate rectangular image region detected by the object detector 31. Refer to Figures 8(a) and 8(b). For example, the feature calculation unit 32 may include a region estimator that estimates the segment region M of an object captured in the candidate rectangular image region, and a part estimator that estimates the coordinates of keypoints KP (joint points such as the head, hands, and feet). These region estimators and part estimators may be learning models implemented using neural networks or the like.

[0056] For example, the feature calculation unit 32 may calculate image features of the type set by the operator for these candidate rectangular image regions, segment regions M, and keypoints KP as candidate features. For example, the feature calculation unit 32 may calculate the shape features of the candidate rectangular image regions and segment regions M, or the positional relationships between keypoints KP as candidate features.

[0057] Alternatively, the feature calculation unit 32 may perform image processing on the input image Im to generate multiple types of feature images (e.g., edge intensity images, edge binarized images, line detection images) that emphasize the features of the input image Im, and may also determine the calculation ranges for multiple types of features (e.g., the central region of the candidate rectangular image region, the central region of the segment region M, the region around the head) based on the positions of the candidate rectangular image region, the segment region M, and the keypoint KP.

[0058] For example, the feature calculation unit 32 may calculate multiple types of candidate features by calculating multiple types of statistics (e.g., mean, variance, binarized pixel count) in multiple calculation ranges for each of the multiple types of feature images. For example, (N1 × N2 × N3) types of candidate features may be calculated by calculating N3 types of statistics in N2 calculation ranges for each of the N1 types of feature images. The feature calculation unit 32 outputs the calculated multiple types of candidate features to the feature selection unit 33. At this time, the feature calculation unit 32 may output the candidate features calculated for each candidate rectangular image region to the feature selection unit 33, associating them with the candidate rectangular image region.

[0059] The feature selection unit 33 selects from among multiple types of candidate features calculated by the feature calculation unit 32 that are likely to separate the low-scoring image region into a low-scoring correct image region IcL and a low-scoring incorrect image region IwL. In other words, the feature selection unit 33 selects a combination of feature types that are likely to separate the low-scoring image region into a low-scoring correct image region IcL and a low-scoring incorrect image region IwL. In the following explanation, the multiple types of features selected by the feature selection unit 33 may be referred to as "separability features".

[0060] At this time, the feature selection unit 33 extracts a low-scoring correct image region IcL and a low-scoring incorrect image region IwL from the candidate rectangular image regions detected by the object detector 31. For example, the feature selection unit 33 may select only low-scoring image regions from the candidate rectangular image regions detected by the object detector 31 based on the scores input from the object detector 31 for each candidate rectangular image region. Furthermore, as described above, the input image Im is an annotated image and is provided with positional information of objects (people) depicted on the input image Im, so the feature selection unit 33 can determine whether each low-scoring image region is a low-scoring correct image region IcL or a low-scoring incorrect image region IwL.

[0061] The feature selection unit 33 selects multiple types of separable features in a feature space with features as axes, such that the distribution of the first image region group consisting of low-scoring correct image regions IcL and the distribution of the second image region group consisting of low-scoring incorrect image regions IwL are easily separated. The first image region group and the second image region group are examples of the "first data region group" and "second data region group" described in the claims. Figure 9 is a schematic diagram showing the distribution of low-scoring ground truth image regions IcL and low-scoring incorrect image regions IwL in the feature space FS, with multiple features F1 and F2 as axes, respectively. The circular and triangular plots indicate the positions of the feature vectors for the low-scoring ground truth image regions IcL and low-scoring incorrect image regions IwL, respectively. The same applies to Figures 13(a) to 13(c).

[0062] In the example of the feature space FS in Figure 9, the low-scoring ground truth image region IcL (circular plot) is concentrated in the range where feature F1 is less than or equal to threshold th1 and feature F2 is less than or equal to threshold th2. Therefore, the feature selection unit 33 selects a combination of feature F1 and feature F2 as separable features that makes it easy to separate the distribution of the first image region group consisting of the low-scoring ground truth image region IcL from the distribution of the second image region group consisting of the low-scoring incorrect image region IwL. In other words, the feature selection unit 33 selects combinations of feature types F1 and F2 that have high separability between the distribution of the first image region group and the distribution of the second image region group in the feature space FS (high discriminability between the distributions of the first image region group and the second image region group) as multiple types of separable features.

[0063] For example, the feature selection unit 33 may use an adaptive boosting algorithm to select types of separable features that make it easier to separate the first image region group from the second image region group. Specifically, the feature selection unit 33 may select a combination of separable feature types by performing the following steps 1 to 3. Step 1 initializes the data weights, and then steps 2 and 3 are repeated a predetermined number of times. In step 2, the feature selection unit 33 sequentially selects the types of separable features. (Step 1) First, the feature selection unit 33 selects the candidate rectangular image region y input to the feature calculation unit 32. j Weight coefficient w j Set the same value for the weight coefficient w. j Assume that all data has been normalized so that the sum of all data equals 1.

[0064] (Step 2) The feature selection unit 33 searches for a candidate feature and its threshold that best distinguishes the low-scoring correct image region IcL from the low-scoring incorrect image region IwL among the candidate features calculated by the feature calculation unit 32. For example, the feature selection unit 33 sets multiple candidate thresholds thc for each of the multiple candidate features fc calculated by the feature calculation unit 32. Then, based on the comparison results between candidate feature fc and candidate thc, the candidate rectangular image region y j A weak classifier h that identifies whether or not a given image is a low-scoring ground truth image region IcL. k (y j ) defines. The feature selection unit 33 uses the error function ε obtained by the following equation (3). k The weak classifier h that minimizes this value k (y j Explore ).

[0065]

number

[0066] t j is, y j This value represents whether the image region is a low-scoring correct image region or a low-scoring incorrect image region, and the indicator function I(h k (y j )≠t j ) is a weak classifier h k (y j ) is t j It does not match, that is, weak classifier h k (y jThe classifier h has a value of "1" if the identification result is not correct. k (y j ) is t j It has a value of "0" if it matches, i.e., is the correct answer. Furthermore, the feature selection unit 33 calculates the error function ε based on the following equation (4). k The weak classifier h that minimizes this value k (y j ) weight coefficient α k The feature selection unit 33 calculates the error function ε obtained from equation (3) above. k The weak classifier h that minimizes this value k (y j The types of candidate features used in ) are defined as the types of separable features, and the weak classifier h k (y j ), weight coefficient α k Add it along with the other items.

[0067]

number

[0068] (Step 3) The weak classifier h selected in Step 2 k (y j Candidate rectangular image region y ) could not distinguish between the low-scoring correct image region IcL and the low-scoring incorrect image region IwL. j The weight coefficient w j We increase it based on the following equation (5). w j =w j × exp(α k ) …(5) Also, weak classifier h k (y j Candidate rectangular image regions y ) were able to distinguish between the low-scoring correct image region IcL and the low-scoring incorrect image region IwL. j The weight coefficient w j We reduce it based on equation (6). w j =w j × exp(-α) k ) …(6) Subsequently, the weight coefficient w jNormalize the data so that the sum of all data equals 1. Then, repeat step 2 above.

[0069] The feature selection unit 33 repeats steps 2 and 3 a predetermined number of times, and in step 2, selects a weak classifier that minimizes the error function considering the weights of the data at that point, and selects a weak classifier h k (y j The types of features used in ) are defined as the types of separable features, and the weak classifier h k (y j ), weight coefficient α k Along with this, we will add them sequentially.

[0070] The feature selection unit 33 may select the types of separable features using methods other than the adaptive boosting algorithm. For example, a predetermined number of candidate features may be selected as separable features in order of their relative distance from the mean distribution of candidate features in the low-scoring correct image region IcL to the mean distribution of candidate features in the low-scoring incorrect image region IwL. Alternatively, for example, a predetermined number of candidate features may be selected as separable features in order of the smallest number of data points included in the overlapping range between the distribution of candidate features in the low-scoring correct image region IcL and the distribution of candidate features in the low-scoring incorrect image region IwL.

[0071] Refer to Figure 7. The parameter determination unit 34 determines the rule parameters of the rule classifier 13a based on the type of separable feature selected by the feature selection unit 33. For example, the parameter determination unit 34 determines the type of separable feature selected by the feature selection unit 33 as the rule feature F of the rule classifier 13a. i It may be determined as the type. Also, for example, the weak classifier h selected by the feature selection unit 33 in step 2 above. k (y j The threshold used in the above equation (1) is the weak classifier h i (x) threshold th i It may be decided as such.

[0072] For example, the weak classifier h selected in step 2 above. k(y j ) weight coefficient α k The weak classifier h in equation (1) above i (x) weight coefficient α i It may be decided as such. The manual adjustment unit 35 accepts input from an operator who manually adjusts the rule parameters. Rule features F included in the rule parameters i Types and values, threshold th i , weight coefficient α i Among the threshold th i or weight coefficient α i At least one of these may be a variable parameter that can be manually adjusted by an operator. For example, the parameter determination unit 34 determines the weak classifier h selected by the feature selection unit 33. k (y j ) Threshold weight coefficient α i The variable parameter th i , α i This may be set as the initial value. The manual adjustment unit 35 controls the variable parameter th by the operator. i , α i Manual adjustments are acceptable.

[0073] (operation) Figure 10 is a flowchart of an example of the parameter determination method in the first embodiment. In step S1, the image input unit 30 acquires the annotated input image Im stored in the storage unit 22 and inputs it to the object detector 31 and the feature calculation unit 32. In step S2, the object detector 31 detects candidate rectangular image regions and calculates a score for each candidate rectangular image region.

[0074] In step S3, the feature calculation unit 32 calculates multiple types of candidate features. In step S4, the feature selection unit 33 selects a combination of separable feature types from the candidate features calculated by the feature calculation unit 32. In step S5, the parameter determination unit 34 determines the rule parameters of the rule classifier 13a based on the type of separable feature selected by the feature selection unit 33. In step S6, the manual adjustment unit 35 receives input from an operator who manually adjusts the rule parameters. The process then ends.

[0075] (First variation) The candidate rectangular image region may include image regions that are not the target object, even though they have a high score indicating the likelihood that the subject in the candidate rectangular image region is the target object (these may be referred to as "high-scoring incorrect image region IwH" in the following explanation).

[0076] Figure 3(d) is an example of a high-scoring incorrect image region IwH enclosed by rectangle B2 in the input image Im of Figure 2. As shown in Figure 4, the high-scoring incorrect image region IwH has a score of Th2 or higher than the object detection threshold, and the high-scoring image region includes both the high-scoring correct image region IcH and the high-scoring incorrect image region IwH.

[0077] Therefore, if an object is detected as an object within a candidate rectangular image region with a score of Th2 or higher, there is a risk of misidentifying the high-scoring incorrect image region IwH as an object. Therefore, the first modified image recognition device 1 includes a rule classifier that identifies high-scoring incorrect image regions IwH from high-scoring image regions by rule-based identification processing based on determination conditions. A score of Th2 or higher is an example of a "score outside the predetermined range" as described in the claims.

[0078] Figure 11 is a block diagram of an example of the functional configuration of the first modified image recognition device 1. The image recognition device 1 is equipped with a rule classifier 13b in place of the rule classifier 13a of the image recognition device 1 of the first embodiment shown in Figure 5. In the first modified example, the score determination unit 12 outputs high-scoring image regions to the rule classifier 13b and low-scoring image regions to the detection result output unit 14, based on the score output by the object detector 11.

[0079] The rule classifier 13b performs rule-based identification processing on the high-scoring image regions input from the score determination unit 12 to determine whether the input high-scoring image region is a high-scoring correct image region IcH or a high-scoring incorrect image region IwH, and outputs the determination result to the detection result output unit 14.

[0080] The detection result output unit 14 outputs low-scoring image regions as detection results for objects other than the target object. The detection result output unit 14 also determines, based on the judgment result of the rule classifier 13b, whether to output high-scoring image regions as detection results for objects or as detection results for objects other than the target object. For example, the detection result output unit 14 may output high-scoring image regions that the rule classifier 13b has determined to be high-scoring correct image regions IcH as object detection results, and high-scoring image regions that it has determined to be high-scoring incorrect image regions IwH as detection results for non-objects.

[0081] For example, the detection result output unit 14 may correct the score of the high-scoring image region based on the judgment result of the rule classifier 13b, output it as an object detection result if the corrected score is equal to or greater than a predetermined threshold (e.g., object detection threshold Th2), and output it as a non-object detection result if it is less than the predetermined threshold.

[0082] For example, the detection result output unit 14 may reduce the score of the high-scoring incorrect image region IwH by subtracting an offset from the score of the high-scoring incorrect image region IwH or by multiplying it by a correction coefficient of less than 1. The offset and correction coefficient may be fixed values, or they may be determined according to equation (2) above.

[0083] The rule parameters that define the rule-based determination conditions of the rule classifier 13b may be set in the same way as the rule parameters of the rule classifier 13a described above. When setting the rule parameters of the rule classifier 13b, the feature selection unit 33 in Figure 7 selects only the high-scoring image regions from the candidate rectangular image regions detected by the object detector 31 based on the score input from the object detector 31 for each candidate rectangular image region, and then selects several types of separability features from among several types of candidate features calculated by the feature calculation unit 32 from the high-scoring image regions to easily separate the high-scoring correct image region IcH from the high-scoring incorrect image region IwH.

[0084] (Second variation) The image recognition device 1 of the first embodiment shown in Figure 5 and the image recognition device 1 of the first modified example shown in Figure 11 may be combined. Figure 12 is a block diagram of an example of the functional configuration of the image recognition device 1 in the second modified example. The image recognition device 1 includes both a rule classifier 13a and a rule classifier 13b.

[0085] In the first modified example, the score determination unit 12 outputs high-scoring image regions to the rule classifier 13b and low-scoring image regions to the rule classifier 13b based on the score output by the object detector 11. The detection result output unit 14 determines, based on the judgment result of the rule classifier 13b, whether to output the high-scoring image region as the detection result of an object or as the detection result of something other than an object, and determines, based on the judgment result of the rule classifier 13a, whether to output the low-scoring image region as the detection result of an object or as the detection result of something other than an object.

[0086] (Third variation) The feature selection unit 33 of the parameter determination device 20 may stepwise select the types of separable features used as rule features for the rule classifier 13a and the rule classifier 13b. In other words, after selecting a combination of types for the nth separable feature (n=1, 2, ...), the feature selection unit 33 may extract a portion of the low-scoring image regions based on the distribution state of the first image region group in the feature space with the nth type separable feature as the axis, and select a combination of types for the (n+1)th separable feature that makes it easy to separate the extracted low-scoring image regions into the first image region group and the second image region group. The step of selecting a combination of types for the (n+1)th separable feature is an example of the "second feature selection step" described in the claims.

[0087] Referring to Figures 13(a) to 13(c), an example of selecting a combination of separable features in three stages will be explained. Below, an example of selecting the types of separable features to be used as rule features for rule classifier 13a will be explained, but the same selection can be made for rule classifier 13b. In this case, the low-score image region, the low-score correct image region IcL, and the low-score incorrect image region IwL are replaced with the high-score image region, the high-score incorrect image region IwH, and the high-score correct image region IcH, respectively.

[0088] In the first stage shown in Figure 13(a), the feature selection unit 33 selects three types of separable features F1 to F3 for the axes of the first feature space FS1, which are easily separated from the distribution of the first image region group (circular plot) consisting of the low-score correct image region IcL and the second image region group (triangular plot) consisting of the low-score incorrect image region IwL, using all the low-score image regions detected by the object detector 31.

[0089] Next, the feature selection unit 33 determines the range of values ​​Rv1 for the separability features F1 to F3 based on the distribution state of the first image region group in the first feature space FS1. For example, the feature selection unit 33 may select a range where the first image region group is unevenly distributed (for example, a range where low-scoring ground truth image region IcL is concentrated) as the value range Rv1. For example, the value range Rv1 may be selected as a range where the proportion of image regions occupied by the first image region group among the low-scoring image regions is equal to or greater than a threshold.

[0090] In the second stage shown in Figure 13(b), the feature selection unit 33 extracts low-scoring image regions included in the value range Rv1, and selects three types of separability features F4 to F6 for the axes of the second feature space FS2 in which the extracted low-scoring image regions are easily separated into a distribution of the first image region group and a distribution of the second image region group. Next, the feature selection unit 33 determines the range of values ​​Rv2 for the separability features F4 to F6 based on the distribution state of the first image region group in the second feature space FS2.

[0091] In the third stage shown in Figure 13(c), the feature selection unit 33 extracts low-scoring image regions included in the value range Rv2, and selects three types of separability features F7 to F9 for the axes of the third feature space FS3 in which the extracted low-scoring image regions are easily separated into a distribution of the first image region group and a distribution of the second image region group. The value ranges Rv1 and Rv2 may be determined, for example, by defining upper and lower limits for the separability features F1-F3 and F4-F6, or they may be determined by inequalities that use the separability features F1-F3 and F4-F6 as parameters.

[0092] By selecting separable features (i.e., rule features) in multiple stages in this way, the discriminative ability of the rule classifier 13a can be improved compared to selecting the same number of separable features at once in a high-dimensional feature space with the same number of separable features as axes. Figure 14 is a block diagram of an example of the functional configuration of the third modified image recognition device 1.

[0093] The third modified image recognition device 1 includes a plurality of rule classifiers 13s, 13t, and 13u instead of the rule classifier 13a of the image recognition device 1 of the first embodiment. Multiple rule classifiers 13s, 13t, and 13u determine whether an input low-scoring image region is a low-scoring correct image region (IcL) or a low-scoring incorrect image region (IwL) based on judgment conditions defined by rule parameters that are based on separability features selected in multiple stages.

[0094] For example, the rule parameters of rule classifier 13s may be set based on the types of separable features F1 to F3, the rule parameters of rule classifier 13t may be set based on the types of separable features F4 to F6, and the rule parameters of rule classifier 13u may be set based on the types of separable features F7 to F9. Alternatively, the determination criteria for a single rule classifier 13a may be defined and set using rule parameters based on the types of separable features (e.g., features F1 to F9) selected in multiple stages, and the success or failure of the logical AND of these determination criteria may be determined.

[0095] Furthermore, when selecting the types of separable features in multiple stages as described above, the groups of feature types that can be selected as separable features may be restricted at each stage. For example, the first group of feature types that can be selected in the first stage may be different from the second group of feature types that can be selected in the second stage. Similarly, the second group of feature types that can be selected in the second stage may be different from the third group of feature types that can be selected in the third stage.

[0096] For example, the group of feature types that can be selected at each stage may be restricted so that the features that can be selected at a later stage (stage (n+1)) are more detailed than the features that can be selected at an earlier stage (stage n). For example, the separability features that can be selected in the first stage may be global image features of the candidate rectangular image region (e.g., the size and width of the candidate rectangular image region).

[0097] The separation features that can be selected in the second stage may be local sub-features of the candidate rectangular image region (for example, the size of the edges of the feet or the brightness value of the head), and the separation features that can be selected in the third stage may be the relationships between sub-features (for example, the relationship between the number of edges around the feet and the number of edges around the chest). By controlling the selection order of separable features from global features to more detailed features in this way, rule features can be selected in a manner that is easy for workers to understand.

[0098] (Fourth variation) If there are too many types of rule features, it becomes difficult for an operator to adjust the rule parameters by manually operating the adjustment unit 35. Therefore, an upper limit may be set on the number of types of rule features (i.e., the number of types of separability features selected by the feature selection unit 33). For example, when selecting separable features in stages, as in the third modified example, the number of stages for selecting separable features may be limited to three, and the number of separable features that can be selected in each stage may be limited to three or fewer (i.e., the total may be limited to nine or fewer).

[0099] (Second Embodiment) The schematic configuration of the image recognition device 1 of the second embodiment is the same as the configuration shown in Figure 1. The image recognition device 1 of the second embodiment identifies which of a predetermined group of classes the input image belongs to. In the following description, the predetermined group of classes identified by the image recognition device 1 of the second embodiment will be simply referred to as "group of classes". The input image is an example of the "input data" described in the claims, and the image recognition device is an example of the "data recognition device" described in the claims.

[0100] Here, we provide an example of a case where multiple classes include a first class (e.g., a person), a second class (e.g., a car), and a third class (animals other than people) (i.e., when classifying the input image into the first to third classes). Note that the number of classes identified by the image recognition device 1 is not limited to three, and the types of classes are not limited to the above example (person, car, animal). The number of classes identified by the image recognition device 1 may be two, four or more.

[0101] Figure 15 is a block diagram of an example of the functional configuration of the image recognition device 1 according to the second embodiment. The image recognition device 1 includes an image input unit 40, a class classifier 41, a score determination unit 42, a first rule classifier 43a, a second rule classifier 43b, and a third rule classifier 43c, which are rule classifiers provided for each of the multiple classes, and an identification result output unit 44. In the following description, the first rule classifier 43a, the second rule classifier 43b, and the third rule classifier 43c may be collectively referred to as "rule classifier 43". The number of rule classifiers 43 does not necessarily have to be the same as the number of classes, and may be less than the number of classes.

[0102] In Figure 1, the image processing unit 5 functions as a class classifier 41, a score determination unit 42, a first rule classifier 43a, a second rule classifier 43b, and a third rule classifier 43c. Furthermore, the communication unit 3 works in cooperation with the image processing unit 5 to function as an image input unit 40 and an identification result output unit 44. The image input unit 40 sequentially acquires the images captured by the shooting unit 2 and inputs them to the class classifier 41 as input images Im.

[0103] The class classifier 41 has a learning model that generates a score for each of the multiple classes indicating the likelihood that the input image Im belongs to one of the multiple classes. In other words, the learning model generates a score indicating the likelihood that the input image Im belongs to a predetermined class. For example, the learning model may be artificial intelligence (AI) implemented by a neural network or the like, and may be a learning model that has been trained using training image data and ground truth data for the classes of the training image data given to the training image data. For example, the class classifier 41 generates a first score S1 indicating the likelihood that the input image Im belongs to the first class, a second score S2 indicating the likelihood that it belongs to the second class, and a third score S3 indicating the likelihood that it belongs to the third class. The class classifier 41 outputs the scores generated for each of the multiple classes, along with the input image Im, to the score determination unit 42.

[0104] The score determination unit 42 outputs the scores (first score S1 to third score S3) output by the class classifier 41 to the identification result output unit 44. The score determination unit 42 also determines whether each of the scores generated by the class classifier 41 for each of the multiple classes is below a predetermined threshold. For example, the score determination unit 42 determines whether the first score S1 is below a predetermined threshold, whether the second score S2 is below a predetermined threshold, and whether the third score S3 is below a predetermined threshold. A score below a predetermined threshold is an example of a "score outside the predetermined range" as described in the claims.

[0105] The score determination unit 42 outputs the input image Im to the rule classifier 43 corresponding to the class from which a score below a predetermined determination threshold was generated among multiple classes. For example, if the first score S1 is less than a predetermined threshold, the input image Im is output to the first rule classifier 43a. If the second score S2 is less than a predetermined threshold, the input image Im is output to the second rule classifier 43b.

[0106] If the third score S3 is less than a predetermined judgment threshold, the input image Im is output to the third rule classifier 43c. The score determination unit 42 may output the input image Im to multiple rule classifiers 43. For example, if both the second score S2 and the third score S3 are less than a predetermined judgment threshold, the input image Im may be output to the second rule classifier 43b and the third rule classifier 43c.

[0107] The rule classifier 43 identifies the class of an input image by performing rule-based identification processing based on judgment conditions on the input image region. Specifically, it performs rule-based identification processing on the input image Im input from the score judgment unit 42 to determine whether the class of the input image Im is the class corresponding to the rule classifier 43, and outputs the judgment result to the detection result output unit 14.

[0108] For example, the first rule classifier 43a determines whether the input image Im belongs to the first class, the second rule classifier 43b determines whether the input image Im belongs to the second class, and the third rule classifier 43c determines whether the input image Im belongs to the third class. The decision conditions used in the rule-based classification process of the rule classifier 43 are defined by parameters relating to the image features of the input image Im.

[0109] Hereinafter, as in the first embodiment, the parameters that define the rule-based determination conditions of the rule classifier 43 will be referred to as "rule parameters," and the image features related to the rule parameters will be referred to as "rule features." The determination condition may be defined by rule parameters relating to multiple types of rule features. For example, the rule classifier 43 may determine whether the determination condition is met based on the weighted sum of the comparison results between the values ​​of the features included in the combination of multiple types of rule features and the threshold.

[0110] For example, the rule classifier 43 may determine whether the class of the input image Im is the class corresponding to the rule classifier 43 based on the classifier H(x) in equation (1) above. i (x) may be a group of weak classifiers that determine whether the class of the input image x is the class corresponding to the rule classifier 43, based on the rule features of the input image x.

[0111] The identification result output unit 44 corrects the score of the input image Im for each class based on the determination result of the rule classifier 43. For example, if the first rule classifier 43a determines that the class of the input image Im is the first class, the first score S1 may be increased by adding an offset to the first score S1 or by multiplying it by a correction coefficient of 1 or more.

[0112] Similarly, if the second rule classifier 43b determines that the input image Im belongs to the second class, or if the third rule classifier 43c determines that the input image Im belongs to the third class, the second score S2 and the third score S3 may be increased. The offset and correction coefficient may be fixed values ​​or determined according to equation (2) above. The identification result output unit 44 outputs the class with the highest corrected score as the identification result.

[0113] Next, the method for determining the rule parameters in the second embodiment will be described. The schematic configuration of the parameter determination device 20 in the second embodiment is the same as the configuration shown in Figure 6. The memory unit 22 stores various programs and data, as well as input images Im for calculating rule features and generating scores, and inputs and outputs this information to and from the image processing unit 23. The input images Im stored in the memory unit 22 are annotated images, and the operator has provided information about the correct class of the input images Im in advance.

[0114] Figure 16 is a block diagram of an example of the functional configuration of the parameter determination device 20 of the second embodiment. The parameter determination device 20 comprises an image input unit 50, a class classifier 51, a feature quantity calculation unit 52, a feature quantity selection unit 53, a parameter determination unit 54, and a manual adjustment unit 55. The image processing unit 23 in Figure 6 functions as an image input unit 50, a class classifier 51, a feature calculation unit 52, a feature selection unit 53, and a parameter determination unit 54. In addition, the display unit 24 and the operation input unit 25 cooperate with the image processing unit 23 to function as a manual adjustment unit 55.

[0115] The following describes an example of determining the rule parameters of the first rule classifier 43a, which identifies whether the input image belongs to the first class or not. However, the rule parameters of the second rule classifier 43b and the third rule classifier 43c can be determined in the same way. In other words, the rule parameters of the rule classifier 43 are determined for each of the multiple classes.

[0116] The image input unit 50 retrieves the annotated input image Im stored in the storage unit 22 and inputs it to the class classifier 51 and the feature calculation unit 52. The class classifier 51 has a learning model that generates a score for each of the multiple classes indicating the likelihood that the input image Im belongs to each of those classes. For example, the class classifier 51 may have the same learning model as the class classifier 41 shown in Figure 15. The class classifier 51 is an example of the "score acquisition unit" described in the claims. The class classifier 51 outputs the calculated score to the feature selection unit 53.

[0117] The feature calculation unit 52 calculates multiple candidate features from the input image Im, which are candidates for multiple types of rule features. The feature calculation unit 52 may calculate multiple candidate features using a method similar to the calculation method by the feature calculation unit 32 in Figure 7. For example, multiple types of candidate features may be calculated by calculating multiple types of statistics (e.g., mean, variance, binarized pixel count) in multiple types of feature images that emphasize the features of the input image Im (e.g., edge intensity image, edge binarized image, line detection image).

[0118] The feature selection unit 53 selects from among the multiple types of candidate features calculated by the feature calculation unit 52 several types of features that make it easy to separate input images Im that are first-class images but have a low first score S1 from input images Im that are not first-class images but also have a low first score S1. In the following description, the multiple types of features selected by the feature selection unit 53 may be referred to as "separability features". For example, the feature selection unit 53 may select an image from the input images Im with a low first score S1 based on the first score S1 output from the class classifier 51. As described above, the input images Im are annotated images and the class information of the input images Im is provided, so the feature selection unit 53 can determine whether or not the class of the input images Im is the first class.

[0119] The feature selection unit 53 selects a number of separable features such that, in a feature space with features as axes, the distribution of a first image group consisting of input images Im of a first class with a low first score S1 is easily separated from the distribution of a second image group consisting of input images Im of classes other than the first class with a low first score S1. In other words, the feature selection unit 53 selects a combination of separable features such that the distribution of the first image group and the distribution of the second image group are easily separated. The first image group and the second image group are examples of the "first data group" and "second data group" described in the claims. The method for selecting separable features by the feature selection unit 53 is the same as the method for selecting features by the feature selection unit 33 in Figure 7.

[0120] The parameter determination unit 54 determines the rule parameters of the first rule classifier 43a based on the type of separable feature selected by the feature selection unit 53. The manual adjustment unit 35 receives input from an operator who manually adjusts the rule parameters.

[0121] Figure 17 is a flowchart of an example of a parameter determination method in the second embodiment. In step S11, the image input unit 50 retrieves the annotated input image Im stored in the storage unit 22 and inputs it to the class classifier 51 and the feature calculation unit 52. In step S12, the class classifier 51 calculates a score indicating the likelihood that the input image Im belongs to each of several classes.

[0122] In step S13, the feature calculation unit 52 calculates multiple types of candidate features. In step S14, the feature selection unit 53 selects a combination of separability features for each of the multiple classes from among the candidate features calculated by the feature calculation unit 52. In step S15, the parameter determination unit 54 determines the rule parameters of the rule classifier 43 for each of the multiple classes based on the types of separable features selected by the feature selection unit 53. In step S16, the manual adjustment unit 55 receives input from an operator who manually adjusts the rule parameters. The process then ends.

[0123] (Fifth variation) In the second embodiment, the rule classifier 43 determined whether the class of the input image Im whose score for the class corresponding to the rule classifier 43 is less than a predetermined determination threshold is the class corresponding to the rule classifier 43. Alternatively, it may be determined whether the class of the input image Im whose score for the class corresponding to the rule classifier 43 is equal to or greater than a predetermined determination threshold is the class corresponding to the rule classifier 43.

[0124] For example, the first rule classifier 43a may determine whether the class of the input image Im whose score for the first class is equal to or greater than a predetermined judgment threshold is the first class. In this case, a score that is above a predetermined threshold is an example of a "score outside the predetermined range" as described in the claims.

[0125] The identification result output unit 44 reduces the score of the input image Im of the class corresponding to the rule classifier 43 if the rule classifier 43 determines that the class of the input image Im is not the class corresponding to the rule classifier 43. For example, if the first rule classifier 43a determines that the class of the input image Im is not the first class, the identification result output unit 44 reduces the first score S1 of the input image Im. The identification result output unit 44 outputs the class with the highest corrected score as the identification result.

[0126] (Sixth variation) The image recognition device 1 of the second embodiment may be combined with the image recognition device 1 of the fifth modified example described above. In other words, the image recognition device 1 may include a rule classifier 43L of the second embodiment that determines the class of an input image Im having a score below a predetermined judgment threshold, and a rule classifier 43H of the fifth modified example that determines the class of an input image Im having a score equal to or greater than a predetermined judgment threshold. The rule classifier 43L is provided for each of the multiple classes, similar to the first rule classifiers 43a to the third rule classifiers 43c. The same applies to the rule classifier 43H.

[0127] The rule classifier 43L determines whether the class of the input image Im whose score for the class corresponding to the rule classifier 43L is less than a predetermined judgment threshold is the class corresponding to the rule classifier 43L. For example, in the case of a rule classifier 43L corresponding to the first class, it is determined whether the class of the input image Im whose first score S1 is less than a predetermined threshold is the first class. The rule classifier 43H determines whether the class of the input image Im whose score for the class corresponding to the rule classifier 43H is equal to or greater than a predetermined judgment threshold is the class corresponding to the rule classifier 43L.

[0128] The identification result output unit 44 increases the score of the input image Im corresponding to the class of rule classifier 43L if rule classifier 43L determines that the class of the input image Im corresponds to the class of rule classifier 43L. If rule classifier 43H determines that the class of the input image Im does not correspond to the class of rule classifier 43H, the unit decreases the score of the input image Im corresponding to the class of rule classifier 43. The identification result output unit 44 outputs the class with the highest corrected score as the identification result.

[0129] (Seventh variation) The above description illustrates examples of how the present invention can be applied to object detection in image data and class identification of image data. However, the present invention is applicable to various types of data, not just image data. For example, the present invention may be applied to object detection in audio data and class identification of audio data.

[0130] For example, the present invention may be applied to a data recognition device comprising: a target sound detector having a learning model that detects the time period in which a target sound, such as a person's scream, was recorded from time-series audio data; and a rule classifier that identifies the presence or absence of a target sound (scream) in the output of the target sound detector (time-series audio data of the detected time period). Furthermore, the present invention may be applied to a data recognition device comprising, for example, a target speech detector having a learning model for identifying the class of speech data, and a rule classifier for identifying whether or not the speech data belongs to a predetermined class.

[0131] Furthermore, the present invention may be applied to a parameter determination device that determines the rule parameters that define the judgment conditions of these rule classifiers. For example, the parameter determination device may select a combination of types of rule features to be used for the rule parameters from a plurality of candidate features such as the speed of volume change (in the case of human voices, the change is less abrupt than that of machine sounds), the proportion of high-frequency generation time in the detection period (in the case of screams, the high-frequency period is longer), and the total amount of frequency change in the detection period (in the case of human voices, abrupt frequency changes do not persist).

[0132] (Effects of the embodiment) (1) The parameter determination method performs rule-based identification processing based on determination conditions on the data region and determines the parameters that define the determination conditions, which are used in a rule classifier that identifies a predetermined target for detection. The parameter determination method includes a score acquisition step of obtaining scores from a learning model that detects data regions that are likely to be targets for detection from the input data and generates scores indicating the likelihood of the data regions being targets for detection; a feature calculation step of calculating multiple features for data regions for which scores outside a predetermined range have been generated; a feature selection step of selecting a combination of feature types that makes it easy to separate the multiple data regions for which scores outside a predetermined range have been generated into a first data region group consisting of data regions to be detected and a second data region group consisting of data regions that are not to be detected; and a parameter determination step of determining the parameters based on the selected combination of feature types.

[0133] For example, in the feature selection step, a combination of feature types may be selected that makes it easy to separate multiple data regions into a first data region group and a second data region group in a feature space with multiple features as axes. This allows for assistance in recognition processing for data that is unsuitable for the learning model's recognition process, in recognition processing that detects predetermined targets included in the input data.

[0134] (2) The parameter determination method may include a value range determination step of determining the value range of feature types of the first combination in which the first data region group or the second data region group is unevenly distributed in a feature space centered on the first combination of feature types selected in the feature selection step, and a second feature selection step of selecting a second combination of feature types that makes it easy to separate multiple data regions having feature types included in the value range determined in the value range determination step into a first data region group and a second data region group.

[0135] The parameter determination step may determine the parameters based on the combination of feature types selected in the feature selection step and the second feature selection step. By selecting features in multiple stages in this way, the capabilities of the rule classifier can be improved compared to selecting features all at once in a high-dimensional feature space with the same number of features as axes.

[0136] (3) The first group of features that can be selected in the feature selection step and the second group of features that can be selected in the second feature selection step may be determined to be different from the multiple features calculated in the feature calculation step. For example, the features of the first group may be features that represent global features of the data domain, and the features of the second group may be features that represent local features of the data domain. By restricting the number of features that can be selected at each of the multiple stages in this way, and controlling the selection order, features can be selected using a method that is easy for workers to understand.

[0137] (4) The parameter determination method may further include a manual adjustment step that accepts input from an operator to adjust the parameters. The parameters may include at least the types of features included in the combination of feature types described above, and a threshold for determining whether the features satisfy the judgment conditions. The parameter determination step determines at least the types of features among the parameters based on the combination of feature types selected in the feature selection step, and the manual adjustment step may accept input to adjust the threshold. The performance of the rule classifier can be improved by manually adjusting the parameters by a worker.

[0138] (5) The parameter determination method may further include a manual adjustment step that accepts input from an operator to adjust the parameters. The rule classifier may determine the success or failure of the judgment condition based on the weighted sum of the comparison results of each feature and threshold included in the above combination of feature types. The parameters may include at least the type of feature included in the above combination of feature types, the threshold to be compared with the feature, and the weight coefficient of the weighted sum. The parameter determination step determines at least the type of feature among the parameters based on the combination of feature types selected by the feature selection step, and the manual adjustment step may accept input to adjust at least the weight coefficient. The performance of the rule classifier can be improved by manually adjusting the parameters by a worker.

[0139] (6) The parameter determination method performs rule-based identification processing based on a determination condition on the input data and determines parameters that define the determination condition, which are used in a rule classifier that identifies the class of the input data. The parameter determination method includes a score acquisition step of obtaining a score from a learning model that generates a score indicating the likelihood of the input data belonging to a predetermined class; a feature calculation step of calculating multiple features for input data for which a score outside the predetermined range has been generated; a feature selection step of selecting a combination of feature types that makes it easy to separate the multiple input data for which a score outside the predetermined range has been generated into a first data group consisting of input data of a predetermined class and a second data group consisting of input data other than the predetermined class; and a parameter determination step of determining parameters based on the selected combination of feature types.

[0140] For example, in the score acquisition step, scores may be obtained from the learning model for each of several predetermined classes; in the feature calculation step, multiple features may be calculated for each of the several predetermined classes; in the feature selection step, a combination of feature types may be selected for each of the several predetermined classes; and in the parameter determination step, parameters may be determined for each of the several predetermined classes. This allows for assistance in recognition processing that identifies which of several predetermined classes the input data belongs to, specifically for data that is unsuitable for the learning model's recognition process. [Explanation of Symbols]

[0141] 1…Image recognition device, 2…Shooting unit, 3, 21…Communication unit, 4, 22…Storage unit, 5, 23…Image processing unit, 6, 24…Display unit, 7, 25…Operation input unit, 10, 30, 40, 50…Image input unit, 11, 31…Object detector, 12, 42…Score determination unit, 13a, 13b, 13s, 13t, 13u…Rule classifier, 14…Detection result output unit, 20…Parameter determination device, 32, 52…Feature calculation unit, 33, 53…Feature selection unit, 34, 54…Parameter determination unit, 35, 55…Manual adjustment unit, 41, 51…Class classifier, 43a…First rule classifier, 43b…Second rule classifier, 43c…Third rule classifier, 44…Identification result output unit

Claims

1. A parameter determination method for determining parameters that define the determination conditions, used in a rule classifier that performs rule-based identification processing based on determination conditions on a data area and identifies a predetermined target for detection, A score acquisition step involves detecting data regions that appear to be the target of detection from the input data and obtaining the score from a learning model that generates a score indicating the likelihood of the said data region being the target of detection, A feature calculation step of calculating multiple features for the data region in which the score outside the predetermined range was generated, A feature selection step of selecting a combination of feature types that makes it easy to separate the plurality of data regions from which scores outside the predetermined range have been generated into a first data region group consisting of the data regions to be detected and a second data region group consisting of the data regions not to be detected. A parameter determination step in which the parameters are determined based on the selected combination of feature types, A parameter determination method characterized by including the following.

2. The parameter determination method according to claim 1, characterized in that the feature selection step selects a combination of feature types that makes it easy to separate the plurality of data regions into a first data region group and a second data region group in a feature space with the plurality of features as axes.

3. A value range determination step that determines the value range of feature quantities of the first combination type in the feature space centered on the first combination of feature types selected in the feature selection step, in which the first data region group or the second data region group is unevenly distributed. A second feature selection step involves selecting a second combination of feature types that makes it easy to separate the plurality of data regions having feature quantities included in the value range determined in the value range determination step into a first data region group and a second data region group, Includes, The parameter determination method according to claim 1, characterized in that the parameter determination step determines the parameters based on the combination of types of features selected by the feature selection step and the second feature selection step.

4. The parameter determination method according to claim 3, characterized in that, among the plurality of features calculated in the feature calculation step, the first group of features that can be selected in the feature selection step and the second group of features that can be selected in the second feature selection step are different, wherein the first group and the second group are limited in this way.

5. The parameter determination method according to claim 4, characterized in that the features of the first group are features that represent the global features of the data domain, and the features of the second group are features that represent the local features of the data domain.

6. The process further includes a manual adjustment step that accepts input from an operator to adjust the aforementioned parameters, The parameter includes at least the type of feature included in the combination of feature types, and a threshold for determining whether the feature satisfies the determination condition. The parameter determination step determines at least the type of feature among the parameters based on the combination of feature types selected in the feature selection step, The manual adjustment step includes receiving an input to adjust the threshold, The parameter determination method according to feature 1.

7. The process further includes a manual adjustment step that accepts input from an operator to adjust the aforementioned parameters, The rule classifier determines whether the judgment condition is met based on the weighted sum of the comparison results of the features included in the combination of feature types and the threshold. The parameter includes at least the type of feature included in the combination of feature types, the threshold to be compared with the feature, and the weight coefficient of the weighted sum. The parameter determination step determines at least the type of feature among the parameters based on the combination of feature types selected in the feature selection step, The manual adjustment step includes receiving an input to adjust the weight coefficient, The parameter determination method according to feature 1.

8. A learning model that detects a data region that is likely to be a predetermined target from input data and generates a score indicating the likelihood of the data region being a target, the learning model being trained using training data and ground truth data of the location of the target on the training data given to the training data, A rule classifier that performs rule-based identification processing based on the determination conditions determined by the parameters determined by the parameter determination method described in any one of claims 1 to 7 for the data region in which a score outside the predetermined range is generated from the detection results of the learning model, A data recognition device characterized by comprising the following features.

9. A parameter determination device for determining parameters that define the determination conditions, used in a rule classifier that performs rule-based identification processing based on determination conditions on a data area and identifies a predetermined target for detection, A score acquisition unit that detects data regions that appear to be the target of detection from the input data and acquires the score from a learning model that generates a score indicating the likelihood of the said data region being the target of detection, A feature calculation unit calculates multiple features for the data region in which the score outside the predetermined range was generated, A feature selection unit selects a combination of feature types that makes it easy to separate the plurality of data regions from which scores outside the predetermined range have been generated into a first data region group consisting of the data regions to be detected and a second data region group consisting of the data regions not to be detected. A parameter determination unit that determines the parameters based on the selected combination of feature types, A parameter determination device characterized by including the following:

10. A parameter determination method for determining parameters that define the judgment conditions, used in a rule classifier that performs rule-based identification processing on input data based on judgment conditions and identifies the class of the input data, A score acquisition step involves obtaining the score from a learning model that generates a score indicating the likelihood of the input data belonging to a predetermined class, A feature calculation step of calculating multiple features for the input data from which the score outside the predetermined range was generated, A feature selection step of selecting a combination of feature types that makes it easy to separate the plurality of input data for which scores outside the predetermined range have been generated into a first data group consisting of input data of a predetermined class and a second data group consisting of input data other than the predetermined class, A parameter determination step in which the parameters are determined based on the selected combination of feature types, A parameter determination method characterized by including the following.

11. In the score acquisition step, the score is acquired from the learning model for each of the predetermined classes. In the feature calculation step, the plurality of features are calculated for each of the plurality of predetermined classes, In the feature selection step, a combination of feature types is selected for each of the predetermined classes. The parameter determination method according to claim 10, characterized in that the parameter is determined for each of the plurality of predetermined classes in the parameter determination step.

12. A learning model that generates a score indicating the likelihood of input data belonging to a predetermined class, the learning model being trained using training data and ground truth data of the class of the training data given to the training data, A rule classifier that performs rule-based identification processing based on the judgment conditions determined by the parameters determined by the parameter determination method described in claim 10 or 11 for the input data in which a score outside the predetermined range is generated from the detection results of the learning model, A data recognition device characterized by comprising the following features.

13. A parameter determination device used in a rule classifier that performs rule-based identification processing based on a determination condition on input data and identifies the class of the input data, the device determines the parameters that define the determination condition, A score acquisition unit that acquires the score from a learning model that generates a score indicating the likelihood of the input data belonging to a predetermined class, A feature calculation unit calculates multiple features for the input data from which the score outside a predetermined range was generated, A feature selection unit selects a combination of feature types that makes it easy to separate the plurality of input data for which scores outside the predetermined range have been generated into a first data group consisting of input data of a predetermined class and a second data group consisting of input data other than the predetermined class. A parameter determination unit that determines the parameters based on the selected combination of feature types, A parameter determination device characterized by including the following:

Images