Information processing device, information processing method, and computer program

Abstract

[Problem] To detect, from an input image with high accuracy, a region that includes an object. [Solution] This information processing device comprises: a first detection unit for performing detection processing on an input image to detect a region that includes an object; an identification unit for calculating a feature vector on the basis of an image of the region detected through the detection processing, identifying the object on the basis of the feature vector, and acquiring an identification reliability degree that is the degree of reliability of the result of object identification; and a training unit for selecting a detection region for training from a plurality of detection regions that correspond to a plurality of the feature vectors on the basis of a plurality of the identification reliability degrees and training a model for detecting the region that includes the object on the basis of an image of the selected detection region.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a computer program. [Background technology]

[0002] In robots such as mobile robots and domestic robots, it is common to detect an area containing a target object from a captured input image and identify the object from the image of the detected area. For example, if the target is a face, face detection is a process of finding the face area from the image, and face identification is a process of identifying an individual from the image of the face area. For a robot to function effectively, it is necessary to correctly detect the area containing the target object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-013857 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure has been made in consideration of the above-described problems, and aims to detect an area including an object from an input image with high accuracy. [Means for solving the problem]

[0005] The information processing device of the present disclosure includes a first detection unit that performs a detection process to detect an area containing an object in an input image, an identification unit that calculates a feature vector based on an image of the area detected in the detection process, identifies the object based on the feature vector, and obtains an identification reliability that is the reliability of the object identification result, and a learning unit that selects a detection area for learning from a plurality of detection areas corresponding to a plurality of the feature vectors based on a plurality of the identification reliability, and learns a model that detects an area containing the object based on an image of the selected detection area.

[0006] The information processing method disclosed herein performs a detection process to detect an area containing an object in an input image, calculates a feature vector based on an image of the area detected in the detection process, identifies the object based on the feature vector, obtains an identification reliability that is the reliability of the object identification result, selects a detection area for learning from a plurality of detection areas corresponding to a plurality of the feature vectors based on the plurality of identification reliability, and learns a model to detect an area containing the object based on an image of the selected detection area.

[0007] The computer program of the present disclosure causes a computer to execute the following steps: performing a detection process to detect an area containing an object in an input image; calculating a feature vector based on an image of the area detected in the detection process, identifying the object based on the feature vector, and obtaining an identification reliability that is the reliability of the object identification result; and selecting a detection area for learning from a plurality of detection areas corresponding to a plurality of the feature vectors based on a plurality of the identification reliability, and learning a model to detect an area containing the object based on an image of the selected detection area. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of an information processing device according to the present disclosure. [Figure 2] 1A and 1B are diagrams showing examples of input images processed by an information processing device according to the present disclosure. [Figure 3] 1 shows an example of correspondence data including feature vectors, detection confidences, and identification confidences. [Figure 4] FIG. 10 is a diagram showing an example of a feature space. [Figure 5] FIG. 10 is a diagram showing an example of learning data in which labels are assigned to detection regions corresponding to feature vectors. [Figure 6] Another example of training data is shown below. [Figure 7] 10 is a flowchart illustrating an example of a processing flow of an information processing device according to an embodiment of the present disclosure. [Figure 8] FIG. 1 is a diagram showing an example of the hardware configuration of a computer that executes a series of processes of an information processing device according to the present disclosure using a program. DETAILED DESCRIPTION OF THE INVENTION

[0009] 1 is a block diagram of an information processing device 10 according to an embodiment of the present disclosure. The information processing device 10 according to this embodiment includes a sensor unit 110 that captures an image by imaging, an image input unit 100 that receives the image from the sensor unit 110, a basic detection unit 200 that performs object detection processing using a model (basic model), and a personalization detection unit 300 that performs object detection processing using a model (personalization model). The information processing device 10 also includes a classification unit 400 that performs object classification processing based on images of areas detected by the basic detection unit 200 and the personalization detection unit 300, an output unit 500 that outputs the object classification results, and a learning unit 600 that generates a personalization model used in the personalization detection unit 300 by machine learning. The basic detection unit 200 corresponds to a first detection unit according to the present disclosure, and the personalization detection unit 300 corresponds to a second detection unit according to the present disclosure.

[0010] In this embodiment, the information processing device 10 can be mounted on a mobile robot, a household robot, or other mobile device. However, the information processing device 10 may also be mounted on a fixedly installed device or system, such as a fixedly installed monitoring device.

[0011] The sensor unit 110 includes a sensor device such as a camera that captures images. The images may be still images or videos. Images may be captured at regular time intervals, or images may be captured when a predetermined event occurs. Examples of the timing may include a timing instructed by a user operating the device 10, a timing when a microphone detects a sound of a predetermined pattern or a sound above a predetermined volume, or a timing when a human presence sensor detects a person. Alternatively, the timing may be a timing when a light sensor detects light of a predetermined pattern or above a certain level, or other timing.

[0012] The image input unit 100 receives an image from the sensor unit 110 and sends the received image to the basic detection unit 200 and the personalized detection unit 300 as an input image.

[0013] 2 shows an example of an image 1200 acquired by the sensor unit 110. The image 1200 is an image captured inside a home. It shows multiple people 11_1, 11_2, and 11_3 and multiple objects (such as a sofa and a pet robot). The figure also shows information (rectangular frame) indicating an area detected by a detection process performed by a personalization detection unit (described later), and details of this will be described later.

[0014] The basic detection unit 200 and the personalized detection unit 300 perform a process of detecting an area including an object to be detected from an input image. In this embodiment, the object is a part of a human body, more specifically, a human face. However, the object to be detected may be a body part other than a face, a non-human animal (e.g., a pet in the home) or a part thereof, or an object (e.g., furniture in the home) or a part thereof.

[0015] The basic detection unit 200 has a model (basic model) that has been trained in advance using a basic dataset, and performs a detection process for an area containing an object using the basic model. The basic dataset is a dataset containing a large number of images for each object to be detected. For example, if the object is a face, the dataset contains various images of faces. If the object is a face, data containing images other than faces may also be included in the dataset as negative examples, and training may be performed to improve the accuracy of detecting the face area. The basic detection unit 200 is generally expected to perform detection at high speed and low accuracy, and is expected to have many false positives but a high recall. A general detector may also be used as the basic detection unit 200.

[0016] The basic detection unit 200 outputs a set of information indicating the detected area and a detection reliability (detection score). The basic detection unit 200 sends the set of information indicating the detected area and the detection reliability to the classification unit 400. The information indicating the area is, for example, a rectangle (bounding box) surrounding the object. However, the shape of the area is not limited to a rectangle, and may be other shapes such as a circle or a triangle.

[0017] The detection confidence (detection score) indicates the reliability of the detection result, i.e., the reliability of the detected region. The detection confidence may be the score itself output by the region detection algorithm used by the basic detection unit 200, as long as the value correlates with the probability that an object exists within the bounding box. Alternatively, the detection confidence may be a value based on uncertainty, which can be obtained using the method disclosed in Gal, Yarin. Uncertainty in Deep Learning. 2016. That is, in deep learning, a dropout layer that deactivates some data with a certain probability is added to a typical CNN for training, and the standard deviation of multiple dropped-out outputs during inference is obtained. The standard deviation is used as the uncertainty value, and the value obtained by subtracting the uncertainty value from 1.0 is used as the confidence (confidence = 1.0 - uncertainty value).

[0018] The personalization detection unit 300 performs a process of detecting an area including an object using a model (personalization model) generated by the learning unit 600, which will be described later. The personalization model can detect a specific object (for example, the face of a specific individual) with higher accuracy than the basic model used by the basic detection unit 200. The algorithm (type of model, etc.) of the personalization model may be the same as or different from that of the basic detection unit 200. Note that, at a stage when the personalization model has not yet been learned (generated) by the learning unit 600, the processing of the personalization detection unit 300 does not need to be performed.

[0019] The classification unit 400 acquires one or more sets of information (rectangle information) indicating the area detected by the basic detection unit 200 and detection reliability (detection score). The classification unit 400 also acquires one or more sets of information (rectangle information) indicating the area detected by the personalization detection unit 300 and detection reliability (detection score). The classification unit 400 performs classification processing based on each set acquired from the basic detection unit 200 and the personalization detection unit 300. If the object is a human face, the classification processing includes identifying the individual (determining whose face it is) from the image of the detected area.

[0020] Specifically, the classification unit 400 first generates one feature vector for each set from the image included in the detected region (in this example, an image of a rectangular region). When the classification unit 400 uses a CNN (Convolutional Neural Network), the feature vector is a feature vector of any dimension provided as the output of the CNN. The dimension of the feature vector is not limited to a specific dimension, and may be, for example, 512 or 128. When the object is a face, one feature vector is calculated from one person's face image. When similarity is calculated between feature vectors obtained from the same person, a large similarity tends to be calculated. Conversely, when similarity is calculated between feature vectors obtained from different people, a small similarity tends to be calculated. A specific example of similarity is cosine similarity.

[0021] The classification unit 400 classifies the object using the feature vector generated for each set.

[0022] As an example of object identification, a face image is registered in advance for each individual, and a feature vector is calculated from the registered face image and stored in association with personal identification information (such as a user name or name) indicating the individual. The identification unit 400 calculates the distance (similarity) between each registered feature vector and the calculated feature vector, and acquires personal identification information corresponding to the feature vector with the smallest distance (highest similarity). The identification unit 400 determines the individual identified by the acquired personal identification information as the identification result. The identification unit 400 also acquires the reliability of the identification (identification reliability). The identification reliability may be a value that increases as the distance decreases (the similarity increases), or may be defined in some other way.

[0023] Another example of object identification is to use an identification model generated in advance by machine learning. For example, a model (identification model) that estimates personal identification information from feature vectors is generated by machine learning using training data that associates feature vectors with personal identification information. The training data may be obtained by capturing facial images of one or more people to be identified, calculating feature vectors for each image, and associating each person's personal identification information with the feature vector. The identification unit 400 inputs the calculated feature vectors into the identification model to obtain personal identification information and identification reliability. The identification reliability may be based on uncertainty, as described in Chang, Jie et al., Data Uncertainty Learning in Face Recognition. 2020, and is expressed as "reliability = 1.0 - uncertainty." The identification model may be a CNN or another type of regression model.

[0024] The classification unit 400 sends the feature vector calculated for each set acquired from the basic detection unit 200 to the learning unit 600 together with the classification reliability and the detection reliability included in the set.

[0025] The output unit 500 arranges information indicating the area detected by the personalization detection unit 300 and the object identification result on the input image (for example, superimposing it on the input image) and displays them on the screen. For example, a rectangular frame is displayed in the area, and the object identification result (for example, personal identification information) is displayed near the rectangle. If the area is correctly detected, the object to be detected (a face in this example) is included within the rectangle. The output unit 500 may further display information indicating the area detected by the basic detection unit 200 (rectangle information in this example) and the object identification result superimposed on the input image.

[0026] When two or more identical classification results are obtained, the rectangular frame corresponding to the classification result with the higher classification reliability may be displayed preferentially. Also, detection areas with detection reliability below a certain value may not be displayed. Similarly, classification results with classification reliability below a certain value may not be displayed.

[0027] The learning unit 600 learns the personalization model used in the personalization detection unit 300 (learns the model parameters), thereby improving the accuracy of object region detection in the personalization detection unit 300.

[0028] The learning unit 600 includes a clustering calculation unit 610 , a label generation unit 620 , a storage unit 630 , a personalized learning unit 640 , and a storage unit 650 .

[0029] The clustering calculation unit 610 associates the feature vectors, classification reliability, and detection reliability obtained from the classification unit 400 to generate correspondence data, and stores the correspondence data in the storage unit 650. The storage unit 650 also stores correspondence data that associates feature vectors, classification reliability, and detection reliability previously obtained from the classification unit 400. The storage unit 650 is any storage medium such as a memory, an optical disk, or a hard disk.

[0030] 3 shows a schematic example of correspondence data stored in the storage unit 650. In this example, the feature vector is a 512-dimensional feature vector, and although each component is shown as a symbol, it is actually a real number. The identification reliability and detection reliability are each shown as a symbol, but are actually real numbers.

[0031] The clustering calculation unit 610 reads out multiple feature vectors stored in the storage unit 650 and maps the feature vectors to a feature space with the same number of dimensions as the feature vectors. The clustering calculation unit 610 clusters the feature vectors to generate multiple clusters. Each cluster contains (or belongs to) one or more feature vectors. Feature vectors with high similarity tend to be located close to each other in the feature space, and as a result, tend to be classified into the same cluster.

[0032] FIG. 4 shows an example of a feature space in which multiple feature vectors are mapped. Feature vectors acquired in the past and feature vectors acquired this time are mapped in the feature space. Four clusters 1210, 1220, 1230, and 1240 are generated. As an example, cluster 1210 corresponds to the face region of person 11_1 in FIG. 2. Cluster 1220 corresponds to the upper body or the entire body of person 11_2. Cluster 1230 corresponds to the face region of person 11_3. Cluster 1240 corresponds to the region of object 11_4 (since this embodiment is intended to detect a human face, the detection of object 11_4 corresponds to a false detection). If objects other than human faces are also detected, clusters corresponding to other types of objects (for example, a cluster corresponding to a sofa) may exist.

[0033] Cluster 1210 includes feature vectors 1211 to 1216. Cluster 1220 includes feature vectors 1221 to 1226. Cluster 1230 includes feature vectors 1231 to 1235. Cluster 1240 includes feature vector 1241. One feature vector is obtained from one object (one detection target).

[0034] The clustering calculation unit 610 selects feature vectors that satisfy conditions based on the clustering results. Specifically, the clustering calculation unit 610 selects clusters that satisfy predetermined conditions (called personalized learning conditions) from the set of clusters, and selects all feature vectors included in the clusters.

[0035] The label generation unit 620 assigns a label indicating the detection area to the detection area corresponding to the feature vector selected by the clustering calculation unit 610. Feature vectors assigned the same label mean that they belong to the same cluster. Since feature vectors obtained from the same person have a high degree of similarity (are close to each other in the feature space), it can be said that feature vectors belonging to the same cluster are generally likely to have been obtained from the same person, and therefore it can be said that the detection area corresponding to the selected feature vector is likely to be suitable as a learning target. The label generation unit 620 stores the labeled detection area as learning data (teaching data) in the storage unit 630. The storage unit 630 is any storage medium such as a memory, an optical disk, or a hard disk.

[0036] The personalized learning conditions used in the process of selecting clusters (feature vectors) by clustering calculation unit 610 will be described in detail below.

[0037] As a personalized learning condition, a cluster may contain N or more samples (feature vectors) whose detection reliability is equal to or less than a threshold value td and whose identification reliability is equal to or greater than a threshold value tr. This personalized learning condition is effective when intensively learning human faces that cannot be correctly detected by the basic detection unit 200 (low detection accuracy) and detecting object regions by using the basic detection unit 200 and personalized detection unit 300 in a complementary manner.

[0038] An example of feature vector selection using personalized learning conditions will be described using the example in Figure 4. Assume that td = 0.5, tr = 0.5, and N = 6. Here, it is assumed that the basic detection unit 200 and personalized detection unit 300 target human faces.

[0039] Cluster 1210 includes feature vectors 1211 to 1215 acquired in the past and a currently acquired feature vector 1216. For the currently acquired feature vector 1216, the reliability (detection reliability) of the basic detection unit 200 is 0.7, which is greater than td (i.e., high-accuracy detection by the basic detection unit 200), and the identification reliability is 0.9, which is greater than tr (high identification accuracy due to the presence of person 11_1 nearby). Therefore, for feature vector 1216, the condition "detection reliability is less than td and identification reliability is greater than tr" is not satisfied. Therefore, although the number of samples in cluster 1210 reaches N, cluster 1210 does not satisfy the personalized learning conditions.

[0040] Cluster 1220 includes previously acquired feature vectors 1221 to 1225 and a currently acquired feature vector 1226. For feature vector 1226, the reliability (detection reliability) of the basic detection unit 200, 0.4, is smaller than td (i.e., the detection accuracy of the basic detection unit 200 is low), and the identification reliability, 0.6, is equal to or greater than tr (person 11_2 is looking slightly downward, but is nearby, so the identification accuracy is high). Therefore, for feature vector 1226, the condition "detection reliability is equal to or less than td and identification reliability is equal to or greater than tr" is satisfied. Also, for previously acquired feature vectors 1221 to 1225, the condition "detection reliability is equal to or less than td and identification reliability is equal to or greater than tr" is satisfied. Therefore, cluster 1220 satisfies the personalized learning condition "the cluster contains N or more samples (feature vectors) whose detection reliability is equal to or less than threshold td and whose identification reliability is equal to or greater than threshold tr."

[0041] Cluster 1230 includes feature vectors 1231 to 1234 acquired in the past and feature vector 1235 acquired this time. For feature vector 1235, the reliability (detection reliability) of the basic detection unit 200, 0.4, is assumed to be less than td (i.e., the detection accuracy of the basic detection unit 200 is low), and the identification reliability, 0.3, is assumed to be less than tr (the person 11_2 is far away and facing sideways, so the identification accuracy is low). Therefore, for feature vector 1235, the condition "detection reliability is less than td and identification reliability is greater than tr" is not satisfied. Furthermore, the number of samples in cluster 1220 does not reach N. Therefore, cluster 1230 does not satisfy the personalized learning conditions.

[0042] Cluster 1240 includes the currently acquired feature vector 1241. For feature vector 1241, the reliability 0.2 of the basic detection unit 200 is smaller than td (the object contained in the detection area is not a human face), and the reliability 0.2 of the classification unit is smaller than tr (the object contained in the detection area is not a human face). Furthermore, the number of samples in cluster 1240 does not reach N. Therefore, cluster 1240 does not satisfy the personalized learning conditions.

[0043] Next, the label generation unit 620 receives the feature vector that satisfies the personalized learning conditions from the clustering calculation unit 610 and labels the detection area corresponding to the feature vector. The label format may be any information that can identify the detection area. For example, a label may be defined by a set of the coordinate of one vertex of a rectangular area, its horizontal width, and its vertical width. A label may also be defined by a set of coordinates of opposing vertices of a rectangular area. If the detection area is circular, the label may be defined by its center and radius. Labels may also be defined by other methods. A labeled detection area has the characteristic that the feature vector calculated from the image of the detection area belongs to a cluster (e.g., the above-mentioned cluster 1220) that satisfies the personalized learning conditions. The label generation unit 620 sends the label of the labeled detection area and other information to the storage unit 630.

[0044] The storage unit 630 receives and stores the labels of the labeled detection regions and the like from the label generation unit 620. The labels of the labeled detection regions and the like are used by the personalized learning unit 640 as learning data (teaching data).

[0045] FIG. 5 shows an example of training data stored in the storage unit 630. This shows an example in which labels are assigned to detection areas corresponding to feature vectors. In this example, the label is defined by the top left coordinate (x_i, y_i), width w_i, height h_i. Instead of the top left coordinate, a center coordinate or other appropriate representative point may be used. Labels may also be defined in other formats, such as a pair of the top left and bottom right coordinates of the area. The data also stores an index, an input image (or identification information such as the file name of the input image), a feature vector, and a cluster name. In the example of FIG. 5, both input images with indexes 1 and 2 are I_1, which means that the detection areas indicated by the labels corresponding to indexes 1 and 2 are included in the same input image. Cluster names A, B, etc. are symbols that identify the clusters. The image of the area indicated by the label in the input image corresponds to the image of the detection area.

[0046] Instead of the input image and label, an image of the detection area may be stored. Figure 6 shows an example of training data in this case. Instead of the input image and label, an image of the detection area (dI_1, dI_2, ...) is stored. A link to the image of the detection area may be stored instead of the image of the detection area itself, and the image of the detection area may be stored in a memory area separate from the training data. Also, an image with a certain margin added around the image of the detection area may be stored. For example, the process of detecting candidate face areas is performed using the image of the detection area as described above, but an image with a margin may be used when fine-tuning only the classification of whether it is a face. As an example of a margin, an area of ​​a certain pixel width surrounding a rectangular area may be used.

[0047] 5 and 6, feature vectors and cluster names are stored, but at least one of the feature vectors and cluster names may be omitted. Also, feature vectors or cluster names can be used for learning. For example, cluster names can be used to make the distribution of occurrence frequencies uniform among clusters.

[0048] The personalization learning unit 640 performs personalization learning based on the learning data stored in the memory unit 630. Specifically, the personalization learning unit 640 acquires the learning data from the memory unit 630 and learns the parameters of the personalization model using images of the detection areas indicated by the labels. The parameters are learned using images of the detection areas with labels as positive examples to improve detection accuracy. For example, if the personalization model is CNN, the weights of the CNN are updated. This type of learning makes it possible to detect face areas that appear frequently but could not be detected by the basic detection unit 200 with high accuracy.

[0049] The personalization learning unit 640 sends the learned results (for example, the updated model or the updated weights) to the personalization detection unit 300 to update the personalization model used in the personalization detection unit 300. This enables the personalization detection unit 300 to perform face detection with higher accuracy for the same person.

[0050] FIG. 2 above shows an example of areas detected by the detection process in the personalization learning unit 640 when the detection target is a face. Area 1110 is detected for person 11_1, area 1120 for person 11_2, area 1130 for person 11_3, and area 1140 for object 11_4. In this example, areas 1110 and 1130 include faces, and the detection accuracy is high. Area 1120 includes not only a face but also the entire upper body, and the detection accuracy is not very high. Area 1140 includes an object rather than a human, and the detection accuracy is low. As personalization learning progresses, it is expected that the face area for person 11_2 will be detected more accurately. It is also expected that object 11_4 will no longer be detected. Here, an example in which the detection target is a face is shown, but similar effects can be obtained when other objects are used as detection targets.

[0051] 7 is a flowchart showing an example of a personalized learning process in the information processing device 10 according to an embodiment of the present disclosure. In this embodiment, an example is shown in which the detection target is a human face, but the present invention is not limited to this.

[0052] The basic detection unit 200 executes face detection processing on the image input from the image input unit 100, and obtains the detection result (information on the detection area) and the reliability of the detection result (detection reliability) (S1001).

[0053] The classification unit 400 executes object (face) classification processing on the image of the detection region in the input image, and obtains a feature vector and classification reliability (S1002).

[0054] The clustering calculation unit 610 maps the feature vector acquired in step S1002 and previously acquired feature vectors into a feature space, and performs clustering of the feature vectors in the feature space (S1003). This generates one or more clusters. Each cluster contains one or more feature vectors.

[0055] Clustering calculation unit 610 determines whether each cluster satisfies the personalized learning condition (S1004). If the personalized learning condition is satisfied, the process proceeds to step S1005, and if the personalized learning condition is not satisfied, the process returns to step S1001.

[0056] Note that clusters that have previously been determined to satisfy the personalized learning conditions may be excluded from the targets of personalized learning. Alternatively, the threshold N for the number of samples (number of feature vectors) for clusters corresponding to faces that have already undergone personalized learning may be increased, and learning may be repeated. Whether a cluster corresponds to a face that has already undergone personalized learning may be determined, for example, by determining whether the position of the center of gravity of the cluster is within a certain distance from the center of gravity of a previous cluster. If they are within the certain distance, both clusters may be determined to correspond to the same face (detection target).

[0057] The label generation unit 620 assigns a label to a feature vector belonging to a cluster that satisfies the personalized learning conditions, enabling identification of a detection area corresponding to the feature vector (S1005). The detection area or an image of the detection area corresponding to the labeled feature vector is used as learning data (teaching data) in personalized learning.

[0058] The personalization learning unit 640 performs personalization learning based on the learning data stored in the storage unit 630, thereby learning a personalization model to be used in the personalization detection unit 300 (S1006).

[0059] As described above, according to this embodiment, the image of the region detected by the basic detection unit 200 is classified by the classification unit 400, and a personalized model is learned based on the image of the region with low detection reliability and high classification reliability. This makes it possible to automatically generate in real time a personalized model for the input image that has improved region detection performance compared to the basic detection unit 200.

[0060] (Variation 1) The personalization learning conditions are not limited to the above examples. Variation 1 of the personalization learning conditions may be that there are N or more samples with an identification reliability of tr or higher. In this case, the detection reliability is not used to determine whether the personalization learning conditions have been met. The personalization learning conditions of variation 1 are expected to enhance the functionality of the personalization detection unit 300, eventually stopping the detection operation of the basic detection unit 200 and making it possible to perform detection using only the personalization detection unit.

[0061] Variation 2 of the personalized learning condition may be such that the average detection reliability is equal to or less than td and the average identification reliability is equal to or greater than tr. Variation 2 of the personalized learning condition is effective when the information processing device 10 has ample memory capacity and there are no restrictions on the number of samples.

[0062] (Variation 2) In the above-described embodiment, the face is mainly targeted as a part of the human body, and the accuracy of detecting the area is improved, but parts of the human body other than the face may also be targeted. Also, non-human animals or specific pets may also be targeted. Other non-living objects, such as furniture in a home, may also be targeted.

[0063] (Variation 3) In the above-described embodiment, the basic detection unit 200 and the personalization detection unit 300 are used together, but if the detection accuracy of the personalization detection unit 300 is sufficiently improved by repeating personalization learning by the learning unit 600, it is not necessary to use the basic detection unit 200. In other words, the function of the basic detection unit 200 may be turned off.

[0064] (Variation 4) The personalization detection unit 300 may be initialized to initialize the parameters of the personalization model. This allows redoing if the accuracy of the personalization detection unit 300 does not improve. This also makes it possible to cope with cases where a robot including this device is transferred to another household or another user.

[0065] (Variation 5) In the above-described embodiment, personalized learning was performed using all detection areas (samples) and images of the detection areas that satisfied the personalized learning conditions. When the number of samples (number of detection areas) is large, the samples may be divided into multiple groups and learning may be performed multiple times. For example, 128 samples may be divided into 16 groups, each containing 8 samples, and personalized learning may be performed 16 times. This has the advantage of reducing the computational load. It is also expected that learning accuracy will improve.

[0066] (Variation 5) The information processing device 10 is not limited to being physically integrated, and some of the components may exist as separate devices on a network and be able to communicate with the information processing device 10. For example, the learning unit 600 (clustering calculation unit, label generation unit, storage unit, and personalization learning unit) may be a device that provides cloud services on a network. For example, the weights of the CNN are updated by calculation in the cloud, and the weights are downloaded to the personalization detection unit 300 and applied.

[0067] [Application example] The following describes application examples of the information processing device 10. Note that the above-described information processing device 10 can also be applied to any of the following systems, devices, methods, etc. of the information processing device 10.

[0068] 8 shows an information processing system 1000 as an example of the hardware configuration of a computer that executes a series of processes of the information processing device 10 of the present disclosure using a program. In the computer, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are interconnected via a bus 1004.

[0069] An input / output interface 1005 is also connected to the bus 1004. To the input / output interface 1005, an input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected.

[0070] The input unit 1006 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 1007 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 1008 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 1009 includes, for example, a network interface. The drive drives removable media such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0071] In a computer configured as above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the storage unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing the program. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as needed.

[0072] The program executed by the computer can be applied by recording it on removable media such as package media, for example. In this case, the program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable media into the drive 1010.

[0073] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 1009 and installed in the storage unit 1008.

[0074] This embodiment is not limited to software processing using a CPU, but may be realized by processing using a GPU (Graphical Processing Unit) and a DSP (Digital Signal Processor) (for example, the Snapdragon Neural Processing Engine or similar technology can be used). The device can also be realized by a chip that executes the processing of this embodiment.

[0075] The steps of the processes disclosed herein do not necessarily have to be performed in the order depicted in the flowcharts. For example, the steps may be performed in a different order than depicted in the flowcharts, or some of the steps depicted in the flowcharts may be performed in parallel.

[0076] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0077] Furthermore, the effects of the present disclosure described in this specification are merely examples, and other effects may also be present.

[0078] The present disclosure can also be configured as follows. [Document name] Claims [Item 1] a first detection unit that performs a detection process to detect an area including an object in an input image; an identification unit that calculates a feature vector based on an image of the area detected in the detection process, identifies the object based on the feature vector, and obtains an identification reliability that is a reliability of the object identification result; a learning unit that selects a detection area for learning from a plurality of detection areas corresponding to a plurality of the feature vectors based on a plurality of the discrimination reliabilities, and learns a model that detects an area including the object based on an image of the selected detection area; An information processing device comprising: [Item 2] the first detection unit acquires a detection reliability that is a reliability of the area detected by the detection process; The learning unit selects the detection area based on the plurality of detection reliabilities. Item 1. An information processing device according to item 1. [Item 3] a second detection unit that performs a detection process to detect an area including the object in the input image or another input image based on the model; 3. The information processing device according to item 1 or 2, comprising: [Item 4] The learning unit clustering a plurality of said feature vectors to generate at least one cluster comprising said feature vectors; selecting a cluster from the at least one cluster based on a plurality of the discrimination confidence measures; Selecting a detection region corresponding to the feature vector included in the selected cluster 4. The information processing device according to any one of items 1 to 3. [Item 5] the first detection unit acquires a detection reliability that is a reliability of the area detected by the detection process; The learning unit selects the cluster based on the plurality of detection reliabilities. Item 5. An information processing device according to item 4. [Item 6] The learning unit selects a cluster in which the number of feature vectors whose detection reliability is less than a first threshold and whose identification reliability is equal to or greater than a second threshold reaches a predetermined number. Item 5. An information processing device according to item 5. [Item 7] The learning unit selects clusters in which the average of the detection reliability is less than a first threshold and the average of the identification reliability is equal to or greater than a second threshold. Item 5. An information processing device according to item 5. [Item 8] The learning unit selects a cluster in which the number of feature vectors whose discrimination reliability is equal to or greater than a second threshold reaches a predetermined number. Item 5. An information processing device according to item 4. [Item 9] an output unit that places information indicating the region detected by the second detection unit in the input image or the other input image; 4. The information processing device according to item 3, [Item 10] The identification unit calculates a feature vector based on an image of the area detected by the second detection unit, and identifies the object based on the feature vector. Item 10. The information processing device according to item 3 or 9. [Item 11] The object is a human body or a part of a human body. 11. The information processing device according to any one of items 1 to 10. [Item 12] The part of the human body is a face Item 12. The information processing device according to item 11. [Item 13] A detection process is performed to detect an area including an object in the input image. calculating a feature vector based on an image of the area detected by the detection process, identifying the object based on the feature vector, and obtaining an identification reliability that is a reliability of the object identification result; A detection area for learning is selected from a plurality of detection areas corresponding to a plurality of the feature vectors based on the plurality of discrimination reliabilities, and a model for detecting an area including the object is learned based on an image of the selected detection area. An information processing method performed by a computer. [Item 14] a step of performing a detection process to detect an area including an object in an input image; calculating a feature vector based on an image of the area detected by the detection process, identifying the object based on the feature vector, and obtaining an identification reliability that is a reliability of the object identification result; selecting a detection area for learning from a plurality of detection areas corresponding to a plurality of the feature vectors based on the plurality of discrimination reliabilities, and learning a model for detecting an area including the object based on an image of the selected detection area; A computer program for causing a computer to execute the above. [Explanation of symbols]

[0079] 11 people 100 Image input unit 200 Basic detection unit (first detection unit) 300 Personalization detection unit (second detection unit) 400 Identification Unit 500 Output Unit 600 Learning Department 610 Clustering Calculation Unit 620 Label Generation Unit 630 Storage section 640 Personalized Learning Department 1001 CPU 1002 ROM 1003 RAM 1004 Bus 1005 Input / Output Interface 1006 Input section 1007 Output section 1008 Storage section 1009 Communications Department 1010 Drive 110 Sensor unit 1200 images 1110, 1120, 1130, 1140 Detected area 1210, 1220, 1230, 1240 Clusters 1211~1216, 1221~1226, 1231~1235, 1241 feature vectors

Claims

1. a first detection unit that performs a detection process to detect an area including an object in an input image; an identification unit that calculates a feature vector based on an image of the area detected in the detection process, identifies the object based on the feature vector, and obtains an identification reliability that is a reliability of the object identification result; a learning unit that selects a detection area for learning from a plurality of detection areas corresponding to a plurality of the feature vectors based on a plurality of the discrimination reliabilities, and learns a model that detects an area including the object based on an image of the selected detection area; An information processing device comprising:

2. the first detection unit acquires a detection reliability that is a reliability of the area detected by the detection process; The learning unit selects the detection area based on the plurality of detection reliabilities. The information processing device according to claim 1 .

3. a second detection unit that performs a detection process to detect an area including the object in the input image or another input image based on the model; The information processing device according to claim 1 ,

4. The learning unit clustering a plurality of said feature vectors to generate at least one cluster comprising said feature vectors; selecting a cluster from the at least one cluster based on a plurality of the discrimination confidence measures; Selecting a detection region corresponding to the feature vector included in the selected cluster The information processing device according to claim 1 .

5. the first detection unit acquires a detection reliability that is a reliability of the area detected by the detection process; The learning unit selects the cluster based on the plurality of detection reliabilities. The information processing device according to claim 4 .

6. The learning unit selects a cluster in which the number of feature vectors whose detection reliability is less than a first threshold and whose identification reliability is equal to or greater than a second threshold reaches a predetermined number. The information processing device according to claim 5 .

7. The learning unit selects clusters in which the average of the detection reliability is less than a first threshold and the average of the identification reliability is equal to or greater than a second threshold. The information processing device according to claim 5 .

8. The learning unit selects a cluster in which the number of feature vectors whose discrimination reliability is equal to or greater than a second threshold reaches a predetermined number. The information processing device according to claim 4 .

9. an output unit that places information indicating the region detected by the second detection unit in the input image or the other input image; The information processing device according to claim 3 , comprising:

10. The identification unit calculates a feature vector based on an image of the area detected by the second detection unit, and identifies the object based on the feature vector. The information processing device according to claim 3 .

11. The object is a human body or a part of a human body. The information processing device according to claim 1 .

12. The part of the human body is a face The information processing device according to claim 11.

13. A detection process is performed to detect an area including an object in the input image. calculating a feature vector based on an image of the area detected by the detection process, identifying the object based on the feature vector, and obtaining an identification reliability that is a reliability of the object identification result; A detection area for learning is selected from a plurality of detection areas corresponding to a plurality of the feature vectors based on the plurality of discrimination reliabilities, and a model for detecting an area including the object is learned based on an image of the selected detection area. An information processing method performed by a computer.

14. a step of performing a detection process to detect an area including an object in an input image; calculating a feature vector based on an image of the area detected by the detection process, identifying the object based on the feature vector, and obtaining an identification reliability that is a reliability of the object identification result; selecting a detection area for learning from a plurality of detection areas corresponding to a plurality of the feature vectors based on the plurality of discrimination reliabilities, and learning a model for detecting an area including the object based on an image of the selected detection area; A computer program for causing a computer to execute the above.

Images