Image processing device, image processing method

Abstract

To provide a technique for enabling learning of a learning model based on sampling images sampled on the basis of sampling probabilities of images of learning targets according to learning proficiency levels of the learning targets.SOLUTION: An image processing device is configured to: obtain a learning proficiency level of a learning target in a learning model; obtain a sampling probability of each image of the plurality of learning targets on the basis of the proficiency levels of the plurality of learning targets; and learn the learning model on the basis of sampling images sampled from images of the plurality of learning targets on the basis of the obtained sampling probabilities.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This invention relates to a learning technique for learning models. [Background technology]

[0002] In recent years, accuracy bias in facial recognition due to attributes such as race and gender has become a concern. Accuracy bias is strongly influenced by imbalances in the training data. In mini-batch learning, which is mainly used as a training method for learning models, the sampling frequency of attribute data with small amounts of data is reduced, making it difficult to improve the learning proficiency of such attributes. Therefore, accuracy bias occurs depending on the amount of data for each attribute. The technology disclosed in Non-Patent Document 1 mitigates accuracy bias by adjusting the loss function according to the learning proficiency for each data set. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Xingkun Xu et al.Consistent Instance False Positive Improves Fairness in Face Recognition.In CVPR 2021. [Overview of the project] [Problems that the invention aims to solve]

[0004] However, the technology disclosed in Non-Patent Document 1 cannot resolve the differences in sampling frequency of mini-batch learning caused by accuracy bias. The present invention provides a technology that enables the training of a learning model based on sampled images sampled based on the sampling probability of the image to be learned according to the learning proficiency of the learning target. [Means for solving the problem]

[0005] An aspect of the present invention relates to a learning model A person or attribute first acquiring means for acquiring a learning proficiency level of a learning target; second acquiring means for acquiring sampling probabilities for images of each of a plurality of learning targets based on proficiency levels of the plurality of learning targets; learning means for training the learning model based on sampled images sampled from the images of the plurality of learning targets in accordance with the sampling probabilities acquired by the second acquiring means comprising 、 The first acquisition means is, Based on the first distance relationship between the feature vector of the target learning object and the representative vectors of the multiple learning objects, or the second distance relationship between the representative vector of the target learning object and the representative vectors of the multiple learning objects, the proficiency level of the target learning object is obtained. The second acquisition means is, The sampling probability of the image to be trained is obtained using a calculation method that yields lower sampling probabilities for higher proficiency levels and higher sampling probabilities for lower proficiency levels. characterized in that Effect of the Invention

[0006] According to the present invention, it is possible to enable training of a learning model based on sampled images sampled in accordance with sampling probabilities of images of learning targets that correspond to learning proficiency levels of the learning targets. Brief Description of the Drawings

[0007] [Figure 1] FIG. 1 is a block diagram showing an example hardware configuration of a system. [Figure 2] FIG. 2 is a block diagram showing an example functional configuration of a learning device. [Figure 3] FIG. 3 is a flowchart of learning processing for a learning model. [Figure 4] FIG. 4 is a flowchart of details of processing in step S102. [Figure 5] FIG. 5 is a flowchart of details of processing in step S103. [Figure 6] FIG. 6 is a block diagram showing an example functional configuration of an inference device. [Figure 7] FIG. 7 is a flowchart of face authentication processing. [Figure 8] FIG. 8 is a block diagram showing an example functional configuration of a learning device. [Figure 9] ​A flowchart of the learning process for a learning model. [Figure 10] A flowchart showing the details of the process in step S503. [Modes for carrying out the invention]

[0008] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0009] [First Embodiment] This embodiment describes an example of an image processing device that acquires the learning proficiency level of the learning target in a learning model, acquires the sampling probability of each image of the multiple learning targets based on the learning proficiency levels of the multiple learning targets, and performs learning of the learning model based on the sampled images sampled from the multiple learning targets based on the sampling probability of each image of the multiple learning targets. Furthermore, this embodiment describes the case in which the learning target is a person's face.

[0010] First, an example of the hardware configuration of the system according to this embodiment will be explained using the block diagram in Figure 1. As shown in Figure 1, the system according to this embodiment has an image processing device 100 and an imaging device 112, and the image processing device 100 and the imaging device 112 are configured to communicate with each other via a wired and / or wireless network 111. An input device 109 and a monitor 110 are also connected to the image processing device 100.

[0011] First, let me explain the imaging device 112. The imaging device 112 captures moving images within a range corresponding to its own imaging direction and field of view, and transmits each frame of the moving image as captured images to the image processing device 100 via the network 111. The imaging device 112 may also capture still images periodically or irregularly, and transmit these captured still images as captured images to the image processing device 100 via the network 111.

[0012] Next, the image processing device 100 will be described. The image processing device 100 can be a computer device such as a PC (personal computer), smartphone, or tablet terminal.

[0013] The CPU 101 executes various processes using computer programs and data stored in the ROM 102 and RAM 103. In doing so, the CPU 101 controls the overall operation of the image processing device 100 and executes or controls the various processes described as being performed by the image processing device 100.

[0014] ROM 102 stores configuration data for the image processing device 100, computer programs and data related to the startup of the image processing device 100, computer programs and data related to the basic operation of the image processing device 100, and so on.

[0015] RAM 103 has an area for storing computer programs and data loaded from ROM 102 and external storage device 104, and an area for storing captured images received from imaging device 112 via communication I / F 107. Furthermore, RAM 103 has a work area used by CPU 101 when executing various processes. In this way, RAM 103 can provide various areas as appropriate.

[0016] The external storage device 104 is a large-capacity information storage device (non-volatile memory) such as a hard disk drive. The external storage device 104 stores the OS (operating system), computer programs and data that cause the CPU 101 to execute or control various processes described as being performed by the image processing device 100. The computer programs and data stored in the external storage device 104 are loaded into the RAM 103 as appropriate according to the control of the CPU 101 and become the target of processing by the CPU 101.

[0017] The external storage device 104 may include optical discs such as flexible disks (FD) or Compact Disks (CDs) that can be attached to or removed from the image processing device 100, as well as magnetic or optical cards, IC cards, memory cards, etc.

[0018] An input device 109 is connected to I / F 105. The input device 109 is a user interface such as a keyboard, mouse, or touch panel, and can input various commands to the CPU 101 by being operated by the user.

[0019] A monitor 110 is connected to the I / F 106. The monitor 110 has an LCD screen or a touch panel screen and can display the processing results from the CPU 101 as images, text, etc. If the monitor 110 has a touch panel screen, it can also accept user input such as touches and swipes. This input is notified to the CPU 101.

[0020] The communication interface 107 is an interface for connecting the image processing device 100 to the network 111, and the image processing device 100 performs data communication with the imaging device 112 on the network 111 via the communication interface 107.

[0021] The CPU 101, ROM 102, RAM 103, external storage device 104, I / F 105, I / F 106, and communication I / F 107 are all connected to the system bus 108. Note that the configuration shown in Figure 1 is just one example of a configuration applicable to the system according to this embodiment, and can be modified as appropriate.

[0022] Figure 2 shows a block diagram illustrating an example of the functional configuration of a learning device that trains a learning model. In this embodiment, we will describe a case in which an image processing device 100 is applied to such a learning device. In this embodiment, we will describe a case in which each functional unit shown in Figure 2 is implemented by software (computer program). In the following, we will describe the functional units shown in Figure 2 as the main processing units, but in reality, the functions of the functional units are realized by the CPU 101 executing the computer program corresponding to the functional unit. Note that one or more of the functional units shown in Figure 2 may be implemented in hardware.

[0023] The learning process of the learning model by the learning device will be explained according to the flowchart in Figure 3. In this embodiment, the case in which Convolutional Neural Networks (CNN) are used as the learning model will be explained, but the following explanation is also applicable to other learning models. In this embodiment, the learning process of a face recognition learning model using the <representative vector method>, which is publicly known from the literature "Jiankang Deng, et. al, ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In CVPR, 2019", will be explained as an example. The <representative vector method> is a face recognition learning method that improves learning efficiency by setting feature vectors for the face images (face images) of each person included in the learning data and using them in combination. A representative vector is, for example, if the learning data contains face images of n people (where n is a natural number greater than or equal to 2), a fully connected layer adjacent to the output layer of the learning model.

[0024]

number

[0025] These are the vectors that make up the structure, and are representative vectors.

[0026]

number

[0027] x is the representative vector corresponding to person ID = j. Person ID is unique identification information for each person. d is the dimension of the representative vector. x is the feature vector obtained (generated) by inputting the face image of person ID = i (the image to be focused on for learning) into the learning model and running the learning model. i teeth

[0028]

number

[0029] This is defined as follows. In this embodiment, during the training of the learning model, the feature vector x is defined as shown in equation (1) below. i and representative vector W j Vector distance based on cosine similarity

[0030]

number

[0031] For example, use the representative vector W. i and feature vector x i The distance between vectors (i.e., the representative vector W of the same person (person ID = i)) i and feature vector x i (Distance between vectors)

[0032]

number

[0033] It takes a smaller value, and the feature vector x of person ID=ii and the sum over j of inter-vector distances between and the representative vector W of the person with person ID=j j

[0034] [Math.]]

[0035] Learning of the learning model is performed by updating parameters (such as weight coefficients) of the learning model such that takes a larger value.

[0036] [Math.]]

[0037] In the present embodiment, the learning proficiency of the learning model for a person with person ID=i is obtained according to the number of representative vectors (other than the representative vector of the person with person ID=i) having a small inter-vector distance from the feature vector obtained from the learning model when the face image of the person with person ID=i is input. Then, the sampling frequency of the face image of each person is controlled such that the sampling frequency of face images of persons with low proficiency becomes high and the sampling frequency of face images of persons with high proficiency becomes low. Then, learning of the learning model is performed using the face images of respective persons sampled according to the controlled sampling frequency.

[0038] In the present embodiment, the learning loop of steps S101 to S104 is repeatedly performed until a learning end condition is satisfied. The learning end conditions include, for example, conditions such as the number of learning times reaching a specified number of times, a specified time having elapsed since the start of learning, a learning error becoming equal to or less than a specified value, and an amount of change in learning error becoming equal to or less than a specified amount. Note that the learning end condition may be a combination of two or more conditions.

[0039] In step S102, face images of respective persons used for learning the learning model are acquired. In step S103, learning of the learning model is performed using the face images of respective persons acquired in step S102. ​

[0040] The details of the process in step S102 will be explained according to the flowchart in Figure 4. In step S201, the acquisition unit 202 acquires the proficiency level obtained by the calculation unit 206 in the previous step S302.

[0041] In step S202, the acquisition unit 202 obtains the probability (sampling probability) of sampling a face image of each of the n people it manages from the training data, based on the "proficiency level of each of the n people". For example, the acquisition unit 202 obtains the proficiency level of person ID=i from N i Therefore, the sampling probability P for person ID=i i We find this according to equation (2) below.

[0042]

number

[0043] Note: Proficiency level N i The sampling probability P of a person with a higher probability i (Sampling frequency) Low To become, proficiency level N i The sampling probability P of a person with a lower probability i (Sampling frequency) high If so, the sampling probability P i The formula for finding is not limited to formula (2) above.

[0044] In the initial step S201, the acquisition unit 202 may acquire and manage pre-set values ​​representing the proficiency levels of each of the n individuals. Furthermore, the acquisition unit 202 uses sampling probabilities P1~P n Alternatively, a predetermined value (a real number between 0 and 1) may be obtained, in which case the processing in step S202 is unnecessary.

[0045] Then, the process in steps S203 to S206 is repeated a specified number of times (for example, a predetermined batch size). In step S204, the acquisition unit 202 sets the sampling probabilities P1 to Pn Based on this, one person ID is selected from 1 to n. For example, suppose n=3 and the sampling probabilities for P1 to P3 are P1=0.3 (30%), P2=0.2 (20%), and P3=0.5 (50%). In this case, the array would be [1, 1, 1, 2, 2, 3, 3, 3, 3, 3], where person ID = i is (number of elements in the array x P). i By generating an array of (1) elements and randomly selecting one from that array, it is possible to select a person ID based on sampling probability.

[0046] In step S205, the acquisition unit 202 selects one face image from the group of face images of the person with the person ID selected in step S204, using the training data (which includes a group of face images of each of the n people) read by the acquisition unit 201 from the external storage device 104.

[0047] Next, the details of the processing in step S103 described above will be explained according to the flowchart in Figure 5. In step S301, the generation unit 203 inputs each face image selected in step S205 (face image of the person with the person ID selected in step S204) into the learning model and obtains the feature vector of the face image obtained by operating the learning model (forward processing). As described above, in this embodiment, a CNN is used as the learning model.

[0048] A Convolutional Neural Network (CNN) extracts abstracted information from an input image by repeatedly performing a set of processing steps—convolution, activation, and pooling—on the input image. The processing unit consisting of convolution, activation, and pooling is often referred to as a hierarchy. Several well-known activation methods exist; for example, the Rectified Linear Unit (ReLU) method may be used. Similarly, several well-known pooling methods exist; for example, the Max Pooling method may be used. For example, the CNN structure may use ResNet, as described in non-patent literature (e.g., K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV, 2016).

[0049] In step S302, the calculation unit 206 uses the feature vectors of each person obtained in step S301 and the representative vectors of each of the n people obtained by the acquisition unit 204 from the external storage device 104 to determine the proficiency level of the person with the person ID selected in step S204. For example, the calculation unit 206 determines the proficiency level N for person ID = i. i We find this according to equation (3) below.

[0050]

number

[0051] Here, T u is the feature vector x i and representative vector W j This is a threshold for determining whether two points are neighbors. I(s) is a function that returns 1 if s is true and 0 if s is false. Note that the following inequality

[0052]

number

[0053] The more j values ​​satisfying this condition, the greater the number of N values. i The smaller the value of N becomes, and the fewer the number of j values ​​that satisfy the inequality, the smaller N becomes. i Any formula that results in a large value of is acceptable, not limited to formula (3) above.

[0054] Furthermore, feature vector x i The more representative vectors whose inter-vector distance is less than the threshold, the greater N i The value of becomes smaller, and the feature vector x i The fewer representative vectors there are whose inter-vector distance is less than the threshold, the smaller N i If the expression makes the value of N large, what kind of expression is used? i You may ask for it.

[0055] Then, in the next step S201, the acquisition unit 202 acquires the "proficiency level of the person with the person ID selected in step S204" obtained in this way, and updates the proficiency level of the person with that person ID in the proficiency levels of the n people it manages with the acquired proficiency level. In this way, the acquisition unit 202 can manage the latest proficiency level for each of the n people.

[0056] In step S303, the update unit 205 updates the parameters of the learning model using the feature vector acquired in step S301 and the representative vector of the person with the person ID selected in step S204 from the representative vectors of each of the n people acquired by the acquisition unit 204 from the external storage device 104, thereby performing the learning process of the learning model, and also saves the representative vectors obtained through this learning to the external storage device 104. As a result, the external storage device 104 always stores the latest representative vectors for each of the n people.

[0057] As a method for updating the parameters of the learning model, for example, a method using a loss function such as ArcFace, as disclosed in the paper "Jiankang Deng, et. al, ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In CVPR, 2019", can be applied.

[0058] Note that while Figure 5 describes the process as being executed in the order of steps S302 and S303, the order of execution of steps S302 and S303 is not limited to this sequence. For example, the processes could be executed in the order of steps S303 and S302, or the processes of step S302 and step S303 could be executed in parallel.

[0059] Next, Figure 6 shows a block diagram illustrating an example of the functional configuration of an inference device that performs inference on the main task using a trained model already trained by the learning device. In this embodiment, the main task is facial recognition, so in the inference for the main task, it is determined whether the registered facial image and the face in the input image belong to the same person.

[0060] The facial recognition process using the inference device will be explained according to the flowchart in Figure 7.

[0061] In step S401, the acquisition unit 301 acquires the user's face image registered in the external storage device 104 and the captured image taken by the imaging device 112. The method and configuration for acquiring the user's face image registered in the external storage device 104 and the captured image taken by the imaging device 112 are not limited to any specific method or configuration.

[0062] For example, the external storage device 104 stores a set of authentication information and a facial image (face image) for each of multiple users. When a user operates the input device 109 and enters their authentication information (user ID, password, etc.), the acquisition unit 301 checks whether the entered authentication information is already registered in the external storage device 104. If the result of this check is that it is already registered, the acquisition unit 301 determines that the first authentication was successful and acquires the facial image associated with the entered authentication information from the external storage device 104. On the other hand, the imaging device 112 is provided to capture the face of the user operating the input device 109, and if the first authentication is successful, the acquisition unit 301 instructs the user to turn their face towards the imaging device 112. The instruction may be notified to the user by displaying a message on the monitor 110 or by voice. After a certain period of time has elapsed since the user operated the input device 109, the imaging device 112 takes an image, and the acquisition unit 301 acquires the image obtained from the image. The method for instructing the imaging device 112 to start imaging is not limited to a specific method.

[0063] In step S402, the generation unit 302 operates in the same manner as the generation unit 203, and obtains the feature vector of the face image obtained by inputting the user's face image acquired by the acquisition unit 301 in step S401 into a trained model by the learning device and operating the trained model. Similarly, the generation unit 302 obtains the feature vector of the captured image obtained by inputting the captured image acquired by the acquisition unit 301 in step S401 into a trained model by the learning device and operating the trained model.

[0064] In step S403, the determination unit 303 calculates the similarity between the feature vector of the face image acquired in step S402 and the feature vector of the captured image acquired in step S402. For example, the determination unit 303 calculates the similarity as the reciprocal of the Euclidean distance between the feature vector of the face image acquired in step S402 and the feature vector of the captured image acquired in step S402.

[0065] The determination unit 303 then determines that if the similarity is above the threshold, the user is the user corresponding to the authentication information, and as a result, the second authentication is deemed successful. On the other hand, if the similarity is below the threshold, the determination unit 303 determines that the user is not the user corresponding to the authentication information, and as a result, the second authentication is deemed unsuccessful.

[0066] In step S404, the output unit 304 outputs the result of the second authentication in step S403. The output format of the second authentication result is not limited to a specific format. For example, the output unit 304 may display the result of the second authentication on the monitor 110 using an image or text. For example, if the second authentication result is successful (the user is the user corresponding to the authentication information), the output unit 304 may display an image or text indicating "Login successful" on the monitor 110. For example, if the second authentication result is unsuccessful (the user is not the user corresponding to the authentication information), the output unit 304 may display an image or text indicating "Login failed" on the monitor 110.

[0067] Alternatively, for example, the output unit 304 may save the result of the second authentication to the external storage device 104. In this case, if the inference device saves the result of the second authentication, which indicates failure, to the external storage device 104 a predetermined number of times consecutively, it may perform processing such as notifying the administrator user of the possibility of unauthorized access.

[0068] For example, the output unit 304 may output the result of the second authentication as audio. Alternatively, for example, the output unit 304 may send a message representing the result of the second authentication in the form of an image or text to an external device via the network 111.

[0069] Thus, according to this embodiment, in mini-batch learning, the level of learning proficiency is calculated, and the sampling probability of images of individuals with low proficiency is increased, thereby reducing the differences in proficiency among individuals, and thus reducing attribute bias in authentication accuracy during inference.

[0070] <Variation> In the first embodiment, proficiency level N i This was calculated using equation (3) above. However, if the training data contains many facial images that are unsuitable for training in facial recognition, such as heavily blurred facial images, it becomes impossible to properly calculate the proficiency level.

[0071] Taking these points into consideration, proficiency level N i Instead of using feature vectors to calculate the proficiency level N, the acquisition unit 204 uses the representative vectors of each of the n individuals acquired from the external storage device 104, and the representative vector of the person with the person ID selected in step S204, as shown in equation (4) below, to calculate the proficiency level N. i You could also try to find that.

[0072]

number

[0073] Furthermore, although the learning device and the inference device were described as separate devices in the first embodiment, the learning device and the inference device may be integrated to form a single processing unit. In this case, the processing unit will have both the functions of a learning device and the functions of an inference device.

[0074] [Second Embodiment] In this embodiment, the differences from the first embodiment will be described, and unless otherwise specified below, it will be assumed to be the same as the first embodiment. An example of the functional configuration of the learning device according to this embodiment is shown in the block diagram of Figure 8. In Figure 8, the same reference numerals are used for functional parts that are the same as those shown in Figure 2, and the description of these functional parts will be omitted.

[0075] In the first embodiment, the sampling probability in mini-batch learning was determined according to the proficiency level of each person. In this embodiment, the facial images of the people are further processed according to their proficiency level. The process of performing processing such as changing the color tone with a certain probability when a facial image used for training is sampled is known as data augmentation and is used as a method to increase the versatility of the learning model. By increasing the probability of performing data augmentation on facial images of people with low proficiency, the versatility can be increased for a variety of facial images, such as those of people with low proficiency, with varying expressions and lighting conditions.

[0076] The learning process of the learning model by the learning device will be explained according to the flowchart in Figure 9. In Figure 9, the same step numbers are used for processing steps that are the same as those shown in Figure 3, and the explanation for these processing steps will be omitted. The flowchart in Figure 9 is a flowchart in which step S503 is inserted between steps S102 and S103 of the flowchart in Figure 3.

[0077] In step S503, for each person ID selected in step S204, the probability of performing Data Augmentation on the face image of that person ID is calculated, and the face image is processed based on this probability. The details of the processing in step S503 will be explained according to the flowchart in Figure 10.

[0078] In step S601, the processing unit 403 obtains the proficiency level that the calculation unit 206 determined in the previous step S302.

[0079] In step S602, the processing unit 403 obtains the probability (processing application probability) of performing Data Augmentation on each face image selected in step S205, based on the "proficiency level of each of the n people" it manages. For example, the processing unit 403 obtains the processing application probability D for performing Data Augmentation on the face image of person ID=i. i This is calculated according to equation (5) below.

[0080]

number

[0081] Here, d is a constant defined by equation (6). Then, the process in steps S603 to S605 is repeated a specified number of times. In step S604, the processing unit 403 selects one unselected face image from the face images selected in step S205 as the selected face image. Then, the processing unit 403 obtains the processing application probability obtained in step S602 for the person in the selected face image, and decides whether or not to apply Data Augmentation to the selected face image based on the processing application probability.

[0082] For example, suppose the face image of person ID=1 is selected as the selected face image, and the probability of applying processing to person ID=1 is obtained as D1=0.3 (30%). In this case, the array [1, 1, 1, 1, 1, 1, 1, 0, 0, 0] has a flag value of "0" indicating that Data Augmentation is applied (the number of elements in the array x D). i An array of ) elements is generated, and one element (flag value) is randomly selected from this array. If the selected flag value is "0", it is decided to apply Data Augmentation; if the selected flag value is "1", it is decided not to apply Data Augmentation. As a method for processing face images to which Data Augmentation has been applied, Cutout, as shown in the literature "Terrance DeVries, et al. Improved Regularization of Convolutional Neural Networks with Cutout.arXiv:1708.04552. 2017", is used.

[0083] The processing unit 403 then applies Data Augmentation to the selected face image if the selected flag value is "0", and does not apply Data Augmentation to the selected face image if the selected flag value is "1".

[0084] In step S605, the processing unit 403 determines whether all of the face images selected in step S205 have been selected as selected face images. If all of the face images selected in step S205 have been selected as selected face images, the loop from steps S603 to S605 ends. On the other hand, if there are any face images remaining from the face images selected in step S205 that have not been selected as selected face images, the process proceeds to step S603.

[0085] Thus, according to this embodiment, by adjusting the probability of applying Data Augmentation to a person's face image according to the learning proficiency level of each person, it is possible to increase versatility for a variety of face images of people with low learning proficiency. This reduces attribute bias.

[0086] In the embodiments and variations described above, the person ID is assumed to be unique identification information for each individual. However, it is not limited to this, and may be unique identification information based on various units such as country, gender, or type.

[0087] Furthermore, the numerical values, processing timings, processing order, processing entity, data (information) destination / source / storage location, etc., used in each of the above embodiments and modifications are given as examples for the purpose of providing specific explanations, and are not intended to limit the scope to such examples.

[0088] Furthermore, some or all of the embodiments and modifications described above may be used in appropriate combinations. Alternatively, some or all of the embodiments and modifications described above may be used selectively.

[0089] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0090] The disclosures herein include the following image processing apparatus, image processing methods, and computer programs.

[0091] (Item 1) A first acquisition method for obtaining the level of proficiency in the learning target in the learning model, A second acquisition means for obtaining the sampling probability of each image of multiple learning targets based on the proficiency level of multiple learning targets, A learning means that trains the learning model based on the sampled images sampled from the plurality of learning target images based on the sampling probability acquired by the second acquisition means, An image processing apparatus characterized by comprising:

[0092] (Item 2) The first acquisition means is, The image processing apparatus according to item 1, characterized in that, among the multiple representative vectors of the target to be studied, the more representative vectors there are in which the distance between the sampled image of the target to be studied and the feature vector obtained based on the learning model is less than a threshold, the lower the proficiency level of the target to be studied, and this is obtained as the proficiency level of the target to be studied.

[0093] (Item 3) The first acquisition means is, The image processing apparatus according to item 1, characterized in that, among the representative vectors of the multiple learning targets, the more representative vectors whose distance from the representative vector of the learning target of interest is less than a threshold, the lower the proficiency level of the learning target of interest.

[0094] (Item 4) The image processing apparatus according to any one of items 1 to 3, characterized in that the second acquisition means acquires a smaller sampling probability as the sampling probability of a learning target with a higher level of proficiency.

[0095] (Item 5) moreover, A third acquisition means for obtaining the probability of performing processing on the sampled image based on the proficiency level of the sampled image, Processing means for performing processing on the sampled image, according to the probability of performing processing on the sampled image. An image processing apparatus according to any one of items 1 to 4, characterized by comprising:

[0096] (Item 6) moreover, An image processing apparatus according to any one of items 1 to 5, characterized by comprising means for performing a second authentication of a person based on a facial image of a person authenticated in the first authentication, an captured image including the person, and a learning model trained by the learning means.

[0097] (Item 7) The information processing device according to item 6, characterized in that the first authentication is authentication using authentication information.

[0098] (Item 8) An image processing method performed by an image processing device, The first acquisition means of the image processing device includes a first acquisition step of acquiring the learning proficiency level of the learning target in the learning model, The second acquisition means of the image processing device includes a second acquisition step of acquiring the sampling probability of each image of a plurality of learning targets based on the proficiency level of the plurality of learning targets, The learning means of the image processing device performs a learning step in which it learns the learning model based on the sampled images sampled from the plurality of images to be learned based on the sampling probability obtained in the second acquisition step. An image processing method characterized by comprising:

[0099] (Item 9) A computer program for causing a computer to function as one of the means of an image processing apparatus described in any one of items 1 to 7.

[0100] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0101] 201: Acquisition unit 202: Acquisition unit 203: Generation unit 204: Acquisition unit 205: Update unit 206: Calculation unit

Claims

1. A first acquisition means for acquiring the learning proficiency level of the learning target, which is a person or attribute in the learning model, A second acquisition means for obtaining the sampling probability of each image of multiple learning targets based on the proficiency level of multiple learning targets, A learning means that performs learning of the learning model based on the sampled images sampled from the plurality of learning target images based on the sampling probability acquired by the second acquisition means, Equipped with, The first acquisition means is, Based on the first distance relationship between the feature vector of the target learning object and the representative vectors of the multiple learning objects, or the second distance relationship between the representative vector of the target learning object and the representative vectors of the multiple learning objects, the proficiency level of the target learning object is obtained. The second acquisition means is, The sampling probability of the image to be trained is obtained using a calculation method that yields lower sampling probabilities for higher proficiency levels and higher sampling probabilities for lower proficiency levels. An image processing apparatus characterized by the following:

2. The first acquisition means is, The image processing apparatus according to claim 1, characterized in that, among the multiple representative vectors of the learning targets, the more representative vectors there are in which the distance between the sampled image of the learning target of interest and the feature vector obtained based on the learning model is less than a threshold, the lower the proficiency level of the learning target of interest, which is obtained as the proficiency level of the learning target of interest.

3. The first acquisition means is, The image processing apparatus according to claim 1, characterized in that, among the representative vectors of the plurality of learning targets, the lower the proficiency of the learning target of interest, the greater the number of representative vectors whose distance from the representative vector of the learning target of interest is less than a threshold, and the lower the proficiency of the learning target of interest.

4. moreover, A third acquisition means for obtaining the probability of performing processing on the sampled image based on the proficiency level of the sampled image, Processing means for performing processing on the sampled image, according to the probability of performing processing on the sampled image. The image processing apparatus according to claim 1, characterized by comprising:

5. The image processing apparatus according to claim 4, characterized in that the processing means generates an array consisting of flag values ​​for processing and flag values ​​for not processing based on the probability of processing the sampled image, and determines whether to perform image processing on the sampled image according to the flag value selected from the array.

6. moreover, The image processing apparatus according to claim 1, comprising means for performing a second authentication of a person based on a facial image of a person authenticated in the first authentication, an image captured including the person, and a learning model trained by the learning means.

7. The image processing apparatus according to claim 6, characterized in that the first authentication is authentication using authentication information indicating an individual entered by the user.

8. The image processing apparatus according to claim 1, characterized in that the learning means generates an array of elements corresponding to the sampling probability of each of the multiple learning targets' images, and performs learning of the learning model based on the image corresponding to the learning target selected from the array.

9. An image processing method performed by an image processing device, The first acquisition means of the image processing device includes a first acquisition step of acquiring the learning proficiency of a learning target which is a person or attribute in the learning model, The second acquisition means of the image processing device includes a second acquisition step of acquiring the sampling probability of each image of a plurality of learning targets based on the proficiency level of a plurality of learning targets, The learning means of the image processing device performs a learning step in which it learns the learning model based on the sampled images sampled from the plurality of images to be learned based on the sampling probability obtained in the second acquisition step. Equipped with, In the first acquisition process described above, Based on the first distance relationship between the feature vector of the target learning object and the representative vectors of the multiple learning objects, or the second distance relationship between the representative vector of the target learning object and the representative vectors of the multiple learning objects, the proficiency level of the target learning object is obtained. In the second acquisition process described above, The sampling probability of the image to be trained is obtained using a calculation method that yields lower sampling probabilities for higher proficiency levels and higher sampling probabilities for lower proficiency levels. An image processing method characterized by the following:

10. A computer program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 8.

Images