Image processing device, image processing method, and image processing program

Abstract

To provide a learning model capable of identifying an attribute of an image with high accuracy.SOLUTION: An image processing apparatus 1 includes an acquisition unit 20A, a pseudo-label estimation unit 20B, and a learning unit 20C. The acquisition unit 20A acquires unsupervised learning data including an image to which a correct label of an attribute is unassigned. The pseudo-label estimation unit 20B estimates a pseudo-label, which is an estimation result of the attribute of the image of the unsupervised learning data, based on an identification target region according to a type of the attribute of an identification target by a first learning model 30 to be learned in the image of the unsupervised learning data. The learning unit 20C learns the first learning model 30 identifying the attribute of the image by using first supervised learning data with the pseudo-label being assigned to the image of the unsupervised learning data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] A technique for training a learning model for identifying image attributes has been disclosed. For example, a technique for training using supervised training data consisting of images to which correct attribute labels have been assigned, and unsupervised training data consisting of images to which correct labels have not been assigned has been disclosed. As a technique using unsupervised training data, a technique for training while estimating attributes of images included in the unsupervised training data has been disclosed. When estimating attributes of images included in the unsupervised training data during training, a technique for estimating and training attributes from the same identification target region as the training model to be trained is used.

[0003] However, depending on the image included in the unsupervised learning data, it may be difficult to estimate attributes from the same classification target region as the learning model being trained. For this reason, conventional techniques may be unable to estimate the attributes of images in the unsupervised learning data, resulting in a decrease in the classification accuracy of the learning model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2012 / 005066 [Non-patent literature]

[0005] [Non-Patent Document 1] Nataniel Ruiz, Eunji Chong, James M. Rehg: Fine-Grained Head Pose Estimation Without Keypoints, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2018, pp. 2074-2083. Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved by the present invention is to provide an image processing device, an image processing method, and an image processing program that can provide a learning model that can identify image attributes with high accuracy. [Means for solving the problem]

[0007] An image processing device according to an embodiment includes an acquisition unit, a pseudo-label estimation unit, and a learning unit. The acquisition unit acquires unsupervised learning data consisting of images to which no correct attribute labels have been assigned. The pseudo-label estimation unit estimates pseudo-labels, which are estimation results of the attributes of the images in the unsupervised learning data, based on a discrimination target region in the images of the unsupervised learning data that corresponds to the type of the attribute to be discriminated by a first learning model to be trained. The learning unit trains the first learning model that discriminates the attributes of the images using first supervised learning data in which the pseudo-labels have been assigned to the images in the unsupervised learning data. When the pseudo label estimation unit determines that it is difficult to estimate the attribute in the image of the unsupervised learning data using a first identification target region, which is the identification target region used for training the first learning model, it estimates the pseudo label based on a second identification target region, which is the identification target region different from the first identification target region. [Brief explanation of the drawings]

[0008] [Figure 1] Schematic diagram of an image processing system. [Figure 2] Schematic diagram of training data. [Figure 3A] Schematic diagram of the image. [Figure 3B] Schematic diagram of the image. [Figure 4] FIG. 10 is an explanatory diagram of a pseudo label estimation process. [Figure 5] FIG. [Figure 6A] An explanatory diagram of learning. [Figure 6B] An explanatory diagram of learning. [Figure 7] 1 is a flowchart of the flow of information processing. [Figure 8] FIG. 10 is an explanatory diagram of a pseudo label estimation process. [Figure 9] 1 is a flowchart of the flow of information processing. [Figure 10] Hardware configuration diagram. DETAILED DESCRIPTION OF THE INVENTION

[0009] The image processing device, image processing method, and image processing program according to the present embodiment will be described in detail below with reference to the accompanying drawings.

[0010] (First embodiment) FIG. 1 is a schematic diagram of an example of an image processing device 1 according to this embodiment.

[0011] The image processing device 1 includes an image processing unit 10, a UI (user interface) unit 14, and a communication unit 16. The image processing unit 10, the UI unit 14, and the communication unit 16 are connected to each other so as to be able to communicate with each other via a bus 18 or the like.

[0012] The UI unit 14 may be configured to be communicably connected to the image processing unit 10 via a wired or wireless connection. The UI unit 14 and the image processing unit 10 may be connected via a network or the like.

[0013] The UI unit 14 has a display function for displaying various information and an input function for receiving user input. The display function is, for example, a display, a projection device, etc. The input function is, for example, a pointing device such as a mouse or a touchpad, a keyboard, etc. The UI unit 14 may be a touch panel that integrates the display function and the input function.

[0014] The communication unit 16 is a communication interface for communicating with an external information processing device or the like of the image processing device 1.

[0015] The image processing device 1 is an information processing device that learns a first learning model 30. The first learning model 30 is a learning model that is a learning target of the image processing device 1. The first learning model 30 is a neural network model for identifying attributes of an image. An attribute is information that represents the properties and characteristics of an image. The first learning model 30 is, for example, a deep neural network (DNN) model obtained by deep learning.

[0016] The image processing unit 10 of the image processing device 1 includes a storage unit 12 and a control unit 20. The storage unit 12 and the control unit 20 are connected to each other via a bus 18 or the like so as to be able to communicate with each other.

[0017] The storage unit 12 stores various types of data. The storage unit 12 may be provided outside the image processing unit 10. Furthermore, at least one of the one or more functional units included in the storage unit 12 and the control unit 20 may be mounted on an external information processing device communicatively connected to the image processing device 1 via a network or the like.

[0018] The control unit 20 executes information processing in the image processing unit 10. The control unit 20 includes an acquisition unit 20A, a pseudo label estimation unit 20B, a learning unit 20C, and an output control unit 20D.

[0019] The acquisition unit 20A, the pseudo label estimation unit 20B, the learning unit 20C, and the output control unit 20D are realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC or circuit, i.e., by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0020] The acquiring unit 20A acquires learning data. The learning data is data used when the first learning model 30 is trained.

[0021] 2 is a schematic diagram of an example of training data 40. The training data 40 includes at least one of supervised training data 42 and unsupervised training data 44.

[0022] The supervised training data 42 is data consisting of images 50 to which correct labels 52 are assigned. The correct labels 52 are labels that represent the attributes of the images 50. In other words, the supervised training data 42 is data consisting of pairs of images 50 and correct labels 52 that represent the attributes of the images 50.

[0023] The unsupervised learning data 44 is data made up of images 50 to which no correct answer label 52 has been assigned. In other words, the unsupervised learning data 44 is data made up of images 50.

[0024] Acquiring unit 20A acquires second supervised learning data 42B and unsupervised learning data 44. Second supervised learning data 42B is an example of supervised learning data 42, and is supervised learning data 42 acquired by acquiring unit 20A.

[0025] The acquiring unit 20A is only required to acquire at least the unsupervised learning data 44 as the training data 40. In this embodiment, an example will be described in which the acquiring unit 20A acquires the unsupervised learning data 44 and the second supervised learning data 42B as the training data 40.

[0026] Returning to Figure 1, we continue the explanation.

[0027] Acquiring unit 20A reads training data 40 from storage unit 12 to acquire unsupervised training data 44 and second supervised training data 42B included in the training data 40. Acquiring unit 20A may also acquire unsupervised training data 44 and second supervised training data 42B included in the training data 40 by receiving training data 40 from an external information processing device or the like via communication unit 16. Acquiring unit 20A may also acquire unsupervised training data 44 and second supervised training data 42B included in the training data 40 by accepting training data 40 input or selected by a user's operation instruction on UI unit 14.

[0028] 3A and 3B are schematic diagrams of an example of an image 50 included in the training data 40. Fig. 3A shows an image 50A. Fig. 3B shows an image 50B. The images 50A and 50B are examples of the image 50.

[0029] In this embodiment, an example will be described in which image 50 is an image including subject S. Subject S may be any of an element captured in image 50 by photographing, or an element created or synthesized by a synthesis process or the like. In other words, image 50 may be any of an image obtained by photographing, an image in which at least a portion of an image obtained by photographing has been synthesized or processed, a synthesized image, a processed image, and a created image.

[0030] In this embodiment, a case where the subject S is a person will be described as an example. Also, in this embodiment, a case where the attribute to be identified by the first learning model 30 is the facial orientation of the subject S will be described as an example. The facial orientation of the subject S is information that indicates the direction in which the face of the subject S is facing. The facial orientation of the subject S is represented, for example, by the angle of the face with respect to a reference direction. The facial orientation of the subject S is represented, for example, by a roll angle, a pitch angle, a yaw angle, etc., with the body axis direction of the subject S, who is a person, as the reference direction.

[0031] In this embodiment, an example will be described in which the first learning model 30 is a learning model that uses a first identification target region 62A included in an image 50 and identifies the attribute of face orientation from the first identification target region 62A.

[0032] The first identification target region 62A is an example of the identification target region 62, and is the identification target region 62 used for learning the first learning model 30. The first identification target region 62A is determined in advance according to the type of attribute to be identified by the first learning model 30. In this embodiment, an example will be described in which the first identification target region 62A is a facial image region of the subject S. The facial image region is a region in the image 50 that represents the face of the subject S, who is a person.

[0033] That is, in this embodiment, an example is described in which the first learning model 30 to be learned is a learning model that takes as input a facial image area, which is the first identification target area 62A contained in the image 50, and outputs the facial direction as an attribute of the image 50.

[0034] The type of attribute may be set in advance according to the target to which the first learning model 30 is applied, and is not limited to the face orientation. The first identification target area 62A may be set in advance according to the type of attribute of the target to be identified by the first learning model 30, and is not limited to the face image area.

[0035] Returning to Figure 1, we continue the explanation.

[0036] The pseudo label estimation unit 20B estimates pseudo labels, which are the estimation results of the attributes of the image 50 of the unsupervised learning data 44, based on an identification target area 62 in the image 50 of the unsupervised learning data 44 that corresponds to the type of attribute to be identified by the first learning model 30.

[0037] First, an outline of the pseudo label estimation process will be described. Hereinafter, the pseudo label estimation process may be referred to as the pseudo label estimation process.

[0038] Fig. 4 is an explanatory diagram showing an example of the flow of the pseudo label estimation process by the pseudo label estimation unit 20 B. Images 50A and 50B shown in Fig. 4 are similar to images 50A and 50B shown in Fig. 3A and 3B, respectively.

[0039] The pseudo label estimation unit 20B estimates pseudo labels 54, which are estimation results of attributes of the images 50 in the unsupervised learning data 44, and generates first supervised learning data 42A.

[0040] First, the acquisition unit 20A acquires training data 40 including unsupervised training data 44 (step S1). The pseudo label estimation unit 20B executes a process of estimating pseudo labels 54 using images 50 included in the unsupervised training data 44 acquired by the acquisition unit 20A.

[0041] The pseudo label estimation unit 20B estimates a pseudo label 54 based on a classification target region 62 that corresponds to the type of attribute of the classification target of the first learning model 30 and that is included in the image 50 of the unsupervised learning data 44. The pseudo label estimation unit 20B predetermines which classification target region 62 in the image 50 is used to estimate the pseudo label 54 when certain estimation conditions are met, depending on the type of attribute of the classification target of the first learning model 30. The estimation conditions will be described later.

[0042] In detail, the pseudo label estimation unit 20B determines whether it is difficult to estimate attributes of the image 50 in the unsupervised learning data 44 using the first classification target region 62A.

[0043] Fig. 4 shows image 50B as an example of image 50 for which it is difficult to estimate attributes using first classification target region 62A, and Fig. 4 also shows image 50A as an example of image 50 for which it is possible to estimate attributes using first classification target region 62A.

[0044] For example, assume that the image 50 included in the unsupervised learning data 44 acquired by the acquisition unit 20A is image 50A (step S2). In image 50A, a first classification target region 62A, which is a facial image region, contains the head of subject S in a state in which the facial orientation can be estimated from the first classification target region 62A. Specifically, the first classification target region 62A of image 50A contains head features such as the eyes, nose, and mouth that are used to estimate the facial orientation. In this case, the pseudo label estimation unit 20B can estimate a pseudo label, which is an estimation result of the facial orientation, from the facial image region, which is the first classification target region 62A of image 50A.

[0045] On the other hand, assume that the image 50 included in the unsupervised learning data 44 acquired by the acquisition unit 20A is image 50B (step S3). Image 50B is an example of an image 50 obtained by capturing a photograph of subject S from the back of the head. In image 50B, the head of subject S is not captured in the first classification target region 62A, which is a facial image region, in a state in which the facial orientation can be estimated from the first classification target region 62A. Specifically, at least some of the head features, such as the eyes, nose, and mouth, used to estimate the facial orientation, are not captured in the first classification target region 62A of image 50B. In this case, it becomes difficult for the pseudo label estimation unit 20B to estimate a pseudo label 54, which is an estimation result of the facial orientation, from the facial image region, which is the first classification target region 62A of image 50A.

[0046] Therefore, when the pseudo label estimation unit 20B determines that it is difficult to estimate an attribute using the first classification target region 62A in the image 50 of the unsupervised learning data 44 (S3), it estimates a pseudo label 54B based on a second classification target region 62B, which is a classification target region 62 different from the first classification target region 62A (step S4). The pseudo label 54B is a pseudo label 54 estimated from the second classification target region 62B and is an example of a pseudo label 54.

[0047] On the other hand, if the pseudo label estimation unit 20B determines that it is possible to estimate an attribute using the first classification target region 62A in the image 50 of the unsupervised learning data 44 (step S2), it estimates a pseudo label 54A based on the first classification target region 62A (step S5). The pseudo label 54A is a pseudo label 54 estimated from the first classification target region 62A and is an example of a pseudo label 54.

[0048] Then, the pseudo label estimation unit 20B generates first supervised training data 42A consisting of pairs of images 50 of the unsupervised training data 44 and estimated pseudo labels 54 (step S6).

[0049] Next, the process of estimating the pseudo label 54 by the pseudo label estimation unit 20B will be described in detail.

[0050] First, the details of the process for determining whether it is difficult to estimate the attribute using the first discrimination target region 62A will be described.

[0051] The pseudo label estimation unit 20B determines whether it is difficult to estimate the attribute using the first classification target region 62A, using a method that depends on the type of attribute to be classified by the first learning model 30 in the image 50 of the unsupervised learning data 44 and the first classification target region 62A.

[0052] For example, the pseudo label estimation unit 20B determines whether the state of the subject S represented by the classification target region 62 in the image 50 of the unsupervised learning data 44 satisfies a predetermined estimation enablement condition.

[0053] The estimable conditions are conditions for estimating an attribute from the first classification target region 62A. In other words, the estimable conditions are conditions used to determine whether or not an attribute can be estimated from the first classification target region 62A.

[0054] The state of the subject S represented by the classification target area 62 and the conditions under which estimation is possible may be determined in advance according to the type of attribute of the classification target by the first learning model 30.

[0055] As described above, in this embodiment, it is assumed that the first classification target area 62A is a face image area of ​​the subject S, and the type of attribute to be classified by the first learning model 30 is face orientation.

[0056] In this case, the pseudo label estimation unit 20B uses, for example, the body angle of the subject S as the state of the subject S represented by the classification target region 62. The body angle is information that represents the orientation of the body of the subject S by an angle. The body angle is represented, for example, by a roll angle, a pitch angle, a yaw angle, etc., with the body axis of the subject S, who is a person, as the reference direction.

[0057] Furthermore, the pseudo label estimation unit 20B uses a predetermined threshold value for the body angle of the subject S as an estimation possible condition. This threshold value may be determined in advance. For example, this threshold value may be determined in advance as a threshold value for distinguishing between a body angle of the subject S in a state where the face direction can be estimated from the face image area and a body angle of the subject S in a state where the face direction is difficult to estimate from the face image area.

[0058] The body angle of the subject S is identified, for example, by detecting the skeleton of the head and other body parts of the subject S. That is, the body angle of the subject S is identified by detecting the skeleton included in an identification target area 62 that is different from the first identification target area 62A, which is the face image area of ​​the subject S. Therefore, in this embodiment, the second identification target area 62B is used as the identification target area 62 used to determine whether the estimable conditions are satisfied.

[0059] The second classification target region 62B is an example of a classification target region 62, and is a classification target region 62 in the image 50 that is different from the first classification target region 62A. The first classification target region 62A and the second classification target region 62B may be classification target regions 62 that differ in at least part of their position, size, and range in one image 50. Furthermore, the first classification target region 62A and the second classification target region 62B may be regions that at least partially overlap within one image 50.

[0060] In this embodiment, an example will be described in which the first identification target region 62A is a face image region, and the second identification target region 62B is a whole body region of the subject S included in the image 50. The whole body region is a region that includes the head and parts other than the head of the subject S. Therefore, the whole body region may be a region that includes the head and at least a part of the area other than the head of the whole body of the subject S, and is not limited to a region that includes the entire body of the subject S, who is a person, from the top of the head to the toes.

[0061] The pseudo label estimation unit 20B identifies a second classification target region 62B, which is the whole body region of the subject S, from the image 50 of the unsupervised learning data 44. A known image processing technique may be used as a method for identifying the second classification target region 62B, which is the whole body region of the subject S, from the image 50. Then, the pseudo label estimation unit 20B detects the skeleton of the subject S from the identified second classification target region 62B, which is the whole body region of the subject S.

[0062] 5 is a diagram illustrating an example of a skeleton detection process performed by the pseudo label estimation unit 20B. An image 50C is shown as an example in FIG. 5. The image 50C is an example of the image 50.

[0063] For example, the pseudo label estimation unit 20B detects the skeletal BG of the subject S from the second identification target region 62B, which is the whole body region of the subject S included in the image 50. A known skeletal detection (human pose estimation) method may be used to detect the skeletal BG of the subject S from the image.

[0064] The pseudo label estimation unit 20B then estimates the body angle of the subject S using information such as the position of one or more parts constituting the body represented by the detected skeleton BG and the angle of one or more joints. A known method may be used to estimate the body angle of the subject S from the detection result of the skeleton BG. The body angle is represented, for example, by a roll angle, a pitch angle, a yaw angle, etc., with the body axis of the subject S, who is a person, as the reference direction.

[0065] Continuing the explanation by returning to Figure 4, if the body angle of the subject S is equal to or greater than the threshold, the pseudo label estimation unit 20B determines that the state of the subject S represented by the second classification target region 62B in the image 50 does not satisfy the estimation enablement condition, and that it is difficult to estimate the attribute using the first classification target region 62A in the image 50 (step S3).

[0066] If the pseudo label estimation unit 20B determines that it is difficult to estimate attributes using the first classification target region 62A in the image 50 of the unsupervised learning data 44 (step S3), it estimates a pseudo label 54B based on the second classification target region 62B (step S4).

[0067] In detail, the pseudo label estimation unit 20B estimates a predetermined pseudo label according to the state of the subject S represented by the second classification target region 62B in the image 50 of the unsupervised learning data 44 (step S4). As described above, in this embodiment, the body angle of the subject S is used as the state of the subject S. Therefore, the pseudo label estimation unit 20B estimates the pseudo label 54B using the body angle of the subject S identified based on the second classification target region 62B, which is the whole body region of the subject S, in the image 50 of the unsupervised learning data 44.

[0068] For example, assume that the angle (e.g., the angle in the yaw direction) represented by the estimated body angle of the subject S is within an angle range representing a person facing backward. In this case, the pseudo label estimation unit 20B estimates "facing backward" as the pseudo label 54B representing the facial orientation, which is an attribute of the image 50.

[0069] The pseudo label estimation unit 20B may pre-store a database associating body angles with pseudo labels 54B, and estimate the pseudo labels 54B by reading the pseudo labels 54B corresponding to the estimated body angles in the database. Alternatively, the pseudo label estimation unit 20B may pre-store a classifier such as a learning model that inputs body angles and outputs pseudo labels 54B, and estimate the pseudo labels using the classifier. It is preferable to use a learning model as the classifier, which has a slower processing speed than the first learning model 30 but outputs classification results with high accuracy.

[0070] In this way, when the pseudo label estimation unit 20B determines that it is difficult to estimate attributes using the first classification target region 62A in the image 50 of the unsupervised learning data 44, it estimates the pseudo label 54B based on the second classification target region 62B (steps S3 and S4).

[0071] Then, the pseudo label estimation unit 20B generates first supervised training data 42A consisting of pairs of images 50 of the unsupervised training data 44 and estimated pseudo labels 54B (step S6).

[0072] On the other hand, if the body angle of the subject S is less than the threshold, the pseudo label estimation unit 20B determines that the state of the subject S represented by the second identification target area 62B in the image 50 satisfies the estimation conditions and that it is possible to estimate attributes using the first identification target area 62A in the image 50 (see steps S2 and S5).

[0073] If the pseudo label estimation unit 20B determines that it is possible to estimate attributes using the first classification target region 62A in the image 50 of the unsupervised learning data 44 (step S2), it estimates a pseudo label 54A based on the first classification target region 62A (step S5).

[0074] Specifically, the pseudo label estimation unit 20B identifies a facial image region, which is a first classification target region 62A, from the image 50 of the unsupervised learning data 44. A known image processing technique may be used to identify the facial image region. The pseudo label estimation unit 20B then estimates a pseudo label 54A from the first classification target region 62A in the image 50 of the unsupervised learning data 44 using the second learning model 32 that has been trained in advance.

[0075] The second learning model 32 is a learning model with a slower processing speed than the first learning model 30.

[0076] That is, the first learning model 30 is a learning model with a faster processing speed than the second learning model 32. Faster processing speed means that the time from inputting an image 50 into the learning model to outputting a classification result is shorter.

[0077] Furthermore, the first learning model 30 is a learning model smaller in size than the second learning model 32. The size of a learning model may be referred to as parameter size. The parameter size is represented by the size of the convolution filter coefficients in the convolution layer of the learning model or the weight size of the fully connected layer. The larger the parameter size, the greater the number of convolution filters, the number of channels of intermediate data output from the convolution layer, and the number of parameters. Therefore, the smaller the learning model, the faster the processing speed, and the larger the learning model, the slower the processing speed. Furthermore, the larger the learning model, the slower the processing speed but the higher the classification accuracy.

[0078] That is, the second learning model 32 is larger in size, has a slower processing speed, and has a larger number of parameters, convolution filters, etc. than the first learning model 30. Therefore, the second learning model 32 is a model that can output more accurate classification results than the first learning model 30, although its processing speed is slower.

[0079] The pseudo label estimation unit 20B inputs a face image region, which is a first discrimination target region 62A, identified from an image 50 included in the unsupervised learning data 44, to the second learning model 32. Then, the pseudo label estimation unit 20B acquires an attribute representing a face direction as an output from the second learning model 32. By acquiring the attribute output from the second learning model 32, the pseudo label estimation unit 20B estimates the attribute as a pseudo label 54A.

[0080] Then, pseudo label estimation unit 20B generates first supervised training data 42A consisting of pairs of images 50 of unsupervised training data 44 and estimated pseudo labels 54A (step S6).

[0081] The explanation will continue by returning to Fig. 1. Next, the learning unit 20C will be explained.

[0082] The learning unit 20C uses the first supervised learning data 42A to learn a first learning model 30 that identifies attributes of images 50 from images 50. The first supervised learning data 42A is learning data 40 in which pseudo labels 54 estimated by the pseudo label estimation unit 20B are assigned to the images 50 of the unsupervised learning data 44.

[0083] As described above, acquiring unit 20A may further acquire second supervised learning data 42B. Therefore, in this embodiment, learning unit 20C may learn first learning model 30 using first supervised learning data 42A and second supervised learning data 42B.

[0084] 6A and 6B are diagrams illustrating an example of learning by the learning unit 20C.

[0085] As shown in FIG. 6A, the learning unit 20C uses the first supervised learning data 42A to which a pseudo label 54 has been assigned and the second supervised learning data 42B to which a correct answer label 52 has been assigned for learning the first learning model 30.

[0086] As shown in FIG. 6B, the learning unit 20C learns a first learning model 30 that outputs an attribute 56, which is a facial orientation, from a first identification target region 62A, which is a facial image region of the image 50, based on an image 50 included in the learning data 40, which is the first supervised learning data 42A or the second supervised learning data 42B, and a pseudo label 54 or a correct label 52 assigned to the learning data 40.

[0087] The learning unit 20C identifies a first classification target region 62A, which is a face image region, from an image 50 included in the learning data 40, and inputs the identified first classification target region 62A to the first learning model 30. Then, the learning unit 20C acquires the attribute 56, which is the face orientation output from the first learning model 30 in response to the input of the first classification target region 62A, as the attribute 56 estimated by the first learning model 30.

[0088] Furthermore, the learning unit 20C learns the first learning model 30 by updating the parameters of the first learning model 30, etc., so as to minimize the least square error L between the attribute 56, which is the face direction estimated by the first learning model 30 from the image 50 included in the learning data 40, and the correct label 52 or pseudo label 54, which is the face direction included in the learning data 40.

[0089] The least square error L is expressed by the following formula (1).

[0090]

number

[0091] In equation (1), L represents the least square error. i (i=1, . . . , N) is the identification information of the training data 40. N is an integer equal to or greater than 2. (x i ,y i ,z i ) is the angle representing the face orientation represented by the pseudo label 54. i is the roll angle, y i is the pitch angle, z i represents the yaw angle. (α i ,β i ,γ i) is the angle representing the face direction output from the first learning model 30. i is the roll angle, β i is the pitch angle, γ i represents the yaw angle.

[0092] Furthermore, when the learning unit 20C uses the correct label 52 of the second supervised learning data 42B, it is sufficient to use the angle representing the face direction represented by the correct label 52B of the second supervised learning data 42B as (xi, yi, zi) in equation (1).

[0093] In addition, the learning unit 20C may perform learning so as to minimize the least square error L using both the pseudo label 54B estimated from the second classification target region 62B and the pseudo label 54A estimated from the first classification target region 62A using the second learning model 32 as the second supervised learning data 42B.

[0094] In this case, the least square error L is expressed by the following equation (2).

[0095]

number

[0096] In equation (2), L represents the least square error. i (i=1, , N) represents the identification information of the training data 40. N is an integer equal to or greater than 2. (α i ,β i ,γ i ) is the angle representing the face direction output from the first learning model 30. i is the roll angle, β i is the pitch angle, γ i represents the yaw angle. (x i ,y i ,z i ) is the angle representing the face orientation represented by the pseudo label 54B estimated from the second classification target region 62B. i is the roll angle, y i is the pitch angle, z i represents the yaw angle.

[0097] In formula (2), (α' i ,β' i ,γ' i ) is the angle representing the face orientation represented by the pseudo label 54A estimated from the first classification target region 62A using the second learning model 32. i is the roll angle, β' i is the pitch angle, γ' i represents the yaw angle. In equation (2), λ is a parameter with a value greater than 0.

[0098] The method of training the first learning model 30 so as to minimize the least square error L expressed by equation (2) is a method called knowledge distillation. By using knowledge distillation, the learning unit 20C can train the first learning model 30 so as to imitate the output of the second learning model 32 that serves as a teacher, and can train the first learning model 30 that can identify attributes with higher accuracy.

[0099] The learning unit 20C may set in advance which of the supervised learning data 42, the first supervised learning data 42A assigned with the pseudo label 54A, and the first supervised learning data 42A assigned with the pseudo label 54B is to be used preferentially for learning. Then, the learning unit 20C may train the first learning model 30 by preferentially using the learning data 40 with a higher priority according to the setting.

[0100] Furthermore, the learning unit 20C may preset a batch size for learning. For example, the learning unit 20C may preset the number of supervised learning data 42, first supervised learning data 42A assigned with pseudo label 54A, and first supervised learning data 42A assigned with pseudo label 54B to be used during learning. The learning unit 20C may then train the first learning model 30 using the number of training data 40 corresponding to the preset numbers.

[0101] The description will continue with returning to Fig. 1. Next, the output control unit 20D will be described.

[0102] The output control unit 20D outputs the first learning model 30 learned by the learning unit 20C. Outputting the first learning model 30 means at least one of displaying information representing the first learning model 30 on the UI unit 14, storing the first learning model 30 in the memory unit 12, and transmitting the first learning model 30 to an external information processing device. For example, the output control unit 20D outputs the first learning model 30 by transmitting the first learning model 30 learned by the learning unit 20C to an external information processing device to which the first learning model 30 is to be applied via the communication unit 16.

[0103] Next, an example of the flow of information processing executed by the image processing unit 10 of this embodiment will be described.

[0104] FIG. 7 is a flowchart showing an example of the flow of information processing executed by the image processing unit 10 of this embodiment.

[0105] The acquiring unit 20A acquires the training data 40 including the second supervised training data 42B and the unsupervised training data 44 (step S100).

[0106] The pseudo label estimation unit 20B determines whether the training data 40 to be processed, among the training data 40 acquired by the acquisition unit 20A, is the second supervised training data 42B to which the correct label 52 has been assigned (step S102).

[0107] If the training data 40 to be processed is the second supervised training data 42B to which the correct label 52 has been assigned (step S102: Yes), the pseudo label estimation unit 20B outputs the second supervised training data 42B to the training unit 20C, and the process proceeds to step S218, which will be described later.

[0108] On the other hand, if the training data 40 to be processed is unsupervised training data 44 to which no correct label 52 has been assigned (step S102: No), the process proceeds to step S104.

[0109] In step S104, the pseudo label estimation unit 20B identifies the second classification target region 62B of the image 50 included in the unsupervised learning data 44 (step S104). That is, the pseudo label estimation unit 20B identifies the second classification target region 62B, which is the whole body region of the subject S included in the image 50.

[0110] The pseudo label estimation unit 20B detects the skeletal BG of the subject S from the second identification target region 62B, which is the whole body region of the subject S identified in step S104 (step S106). Then, the pseudo label estimation unit 20B estimates the body angle of the subject S from the detection result of the skeletal BG detected in step S106 (step S108).

[0111] Next, the pseudo label estimation unit 20B determines whether the body angle estimated in step S108 is less than a threshold value, which is an estimation possible condition (step S110). That is, through the processes of steps S104 to S110, the pseudo label estimation unit 20B determines whether the state of the subject S represented by the classification target region 62 of the image 50 included in the unsupervised learning data 44 satisfies the estimation possible condition for estimating an attribute from the first classification target region 62A.

[0112] If the body angle is less than the threshold value (step S110: Yes), the pseudo label estimation unit 20B determines that it is possible to estimate the face direction using the first discrimination target region 62A, which is a face image region in the image 50. Then, the process proceeds to step S112.

[0113] In step S112, the pseudo label estimation unit 20B estimates a pseudo label 54A from the first discrimination target region 62A and the second learning model 32 (step S112). The pseudo label estimation unit 20B inputs the face image region, which is the first discrimination target region 62A included in the image 50 of the unsupervised learning data 44, to the second learning model 32. Then, the pseudo label estimation unit 20B acquires an attribute representing the face direction as an output from the second learning model 32. By acquiring the attribute output from the second learning model 32, the pseudo label estimation unit 20B estimates the attribute as a pseudo label 54A.

[0114] Then, pseudo label estimation unit 20B generates first supervised training data 42A consisting of pairs of images 50 of unsupervised training data 44 and pseudo labels 54A estimated in step S112 (step S114). Then, the process proceeds to step S120, which will be described later.

[0115] On the other hand, if it is determined in step S110 that the body angle is equal to or greater than the threshold (step S110: No), the pseudo label estimation unit 20B determines that it is difficult to estimate the face orientation using the first classification target region 62A, which is the face image region of the image 50. In other words, if the body angle of the subject S is equal to or greater than the threshold, the pseudo label estimation unit 20B determines that the state of the subject S represented by the second classification target region 62B in the image 50 does not satisfy the estimation possible conditions, and that it is difficult to estimate the attribute using the first classification target region 62A in the image 50. Then, the process proceeds to step S116.

[0116] In step S116, the pseudo label estimation unit 20B estimates a pseudo label 54B from the second classification target region 62B, which is a whole-body region, in the image 50 of the unsupervised learning data 44 (step 116). As described above, for example, the pseudo label estimation unit 20B estimates a pseudo label 54B, such as "facing straight back," using the body angle of the subject S identified based on the second classification target region 62B, which is a whole-body region of the subject S, in the image 50 of the unsupervised learning data 44.

[0117] Then, pseudo label estimation unit 20B generates first supervised training data 42A consisting of pairs of images 50 of unsupervised training data 44 and pseudo labels 54B estimated in step S116 (step S118), and then proceeds to step S120.

[0118] In step S120, the learning unit 20C learns the first learning model 30 using the first discrimination target region 62A included in the learning data 40 (step S120).

[0119] The learning unit 20C accepts the second supervised learning data 42B determined in step S102 (step S102: Yes), the first supervised learning data 42A generated in step S114, and the first supervised learning data 42A generated in step S118 as learning data 40. Then, the learning unit 20C identifies a first classification target region 62A, which is a facial image region, from an image 50 included in the learning data 40, and inputs the first classification target region 62A to the first learning model 30. Then, the learning unit 20C acquires the attribute 56, which is the facial orientation output from the first learning model 30 in response to the input of the first classification target region 62A, as the attribute 56 estimated by the first learning model 30.

[0120] Furthermore, the learning unit 20C learns the first learning model 30 by updating the parameters of the first learning model 30, etc., so as to minimize the least square error L between the attribute 56, which is the face direction estimated by the first learning model 30 from the image 50 included in the learning data 40, and the correct label 52 or pseudo label 54 (pseudo label 54A, pseudo label 54B), which is the face direction included in the learning data 40.

[0121] The output control unit 20D outputs the first learning model 30 learned in step S120 (step S122), and then ends this routine.

[0122] As described above, the image processing device 1 of this embodiment includes an acquisition unit 20A, a pseudo label estimation unit 20B, and a learning unit 20C. The acquisition unit 20A acquires unsupervised learning data 44 consisting of images 50 to which no correct attribute labels 52 have been assigned. The pseudo label estimation unit 20B estimates pseudo labels 54, which are estimation results of attributes of the images 50 in the unsupervised learning data 44, based on a classification target region 62 in the images 50 in the unsupervised learning data 44 that corresponds to the type of attribute to be classified by the first learning model 30 being trained. The learning unit 20C trains the first learning model 30 that recognizes attributes 56 of the images 50 using first supervised learning data 42A in which the pseudo labels 54 have been assigned to the images 50 in the unsupervised learning data 44.

[0123] Here, the prior art discloses a technique for learning while estimating attributes of images 50 included in unsupervised learning data 44. In the prior art, a learning model to be learned is learned while estimating attributes from the same discrimination target region 62 as the learning model to be learned. However, depending on the image included in the unsupervised learning data 44, it may be difficult to estimate attributes from the same discrimination target region 62 as the learning model to be learned. For this reason, the prior art may be unable to estimate attributes of images 50 in the unsupervised learning data 44, resulting in a decrease in the classification accuracy of the learning model to be learned.

[0124] Meanwhile, in the image processing device 1 of this embodiment, the pseudo label estimation unit 20B estimates pseudo labels 54, which are estimation results of attributes of the images 50 of the unsupervised learning data 44, based on a classification target region 62 in the images 50 of the unsupervised learning data 44 that corresponds to the type of attribute to be classified by the first learning model 30 being trained. Then, the learning unit 20C trains the first learning model 30 that classifies attributes 56 of the images 50, using the first supervised learning data 42A in which the pseudo labels 54 are assigned to the images 50 of the unsupervised learning data 44.

[0125] As described above, in this embodiment, the image processing device 1 estimates the pseudo label 54 based on the classification target region 62 according to the type of attribute to be classified by the first learning model 30 to be learned, rather than on a fixed classification target region 62. Then, the image processing device 1 uses the image 50 to which the pseudo label 54 has been assigned as the first supervised learning data 42A to train the first learning model 30.

[0126] Therefore, the image processing device 1 of this embodiment can assign pseudo labels 54 to the unsupervised learning data 44 with high accuracy. Then, the image processing device 1 of this embodiment trains the first learning model 30 using the first supervised learning data 42A to which the pseudo labels 54 have been assigned. Therefore, the image processing device 1 of this embodiment can train the first learning model 30 that can identify the attributes of the image 50 with high accuracy.

[0127] Therefore, the image processing device 1 of this embodiment can provide the first learning model 30 (learning model) that can identify the attributes of the image 50 with high accuracy.

[0128] Furthermore, in the prior art, in order to estimate the attributes of images 50 included in the unsupervised learning data 44 while learning, it was necessary to separately prepare images that did not include a facial image region, which is an attribute to be identified by the first learning model 30, and use them as learning data. In contrast, in the image processing device 1 of this embodiment, the pseudo label estimation unit 20B estimates a pseudo label 54 from images 50 included in the unsupervised learning data 44 based on a classification target region 62 corresponding to the type of attribute to be identified by the first learning model 30. Therefore, the image processing device 1 of this embodiment can train the first learning model 30 without separately preparing images that do not include a facial image region, which is an attribute to be identified by the first learning model 30. Therefore, in addition to the above-mentioned effects, the image processing device 1 of this embodiment can easily train the first learning model 30 with a simple configuration.

[0129] Furthermore, when the pseudo label estimation unit 20B of the image processing device 1 of this embodiment determines that it is possible to estimate an attribute using the first classification target region 62A in the image 50 of the unsupervised learning data 44, it estimates a pseudo label 54A using the first classification target region 62A and the second learning model 32. As described above, the second learning model 32 is a learning model with a slower processing speed than the first learning model 30, but is a model that can output classification results with higher accuracy than the first learning model 30. On the other hand, the first learning model 30 to be learned is a learning model with a faster processing speed than the first learning model 30, but the accuracy of the classification results may be lower than that of the second learning model 32.

[0130] However, the learning unit 20C of the image processing device 1 of this embodiment learns the first learning model 30 using the first supervised learning data 42A to which the pseudo label 54A estimated using the second learning model 32 capable of outputting highly accurate classification results is assigned. Therefore, the learning unit 20C of this embodiment can learn the first learning model 30 that has a high processing speed and can accurately classify the attributes of the image 50.

[0131] (Second embodiment) In this embodiment, an example will be described in which the first learning model 30 to be learned is a learning model that identifies attributes of a different type from those in the above embodiment.

[0132] Note that parts having the same functions or configurations as those in the above-described embodiment may be given the same reference numerals and detailed descriptions thereof may be omitted.

[0133] FIG. 1 is a schematic diagram of an example of an image processing device 1B according to this embodiment.

[0134] The image processing device 1B is similar to the image processing device 1 of the above embodiment, except that it includes an image processing unit 10B instead of the image processing unit 10. The image processing unit 10B is similar to the image processing unit 10 of the above embodiment, except that it includes a control unit 22 instead of the control unit 20. The control unit 22 is similar to the control unit 20 of the above embodiment, except that it includes a pseudo label estimation unit 22B instead of the pseudo label estimation unit 20B.

[0135] In this embodiment, an example will be described in which the attribute to be identified by the first learning model 30 is the gender of the subject S. Also, in this embodiment, as in the above embodiment, an example will be described in which the first identification target region 62A is the facial image region of the subject S. That is, in this embodiment, an example will be described in which the first learning model 30 to be learned is a learning model that receives the facial image region that is the first identification target region 62A of an image 50 as input and outputs the gender of the subject S as an attribute of the image 50.

[0136] In this embodiment, the second discrimination target region 62B, which is a discrimination target region 62 different from the first discrimination target region 62A, is the whole body region of the subject S, as in the above embodiment.

[0137] Similar to the pseudo label estimation unit 20B of the above embodiment, the pseudo label estimation unit 22B estimates a pseudo label 54, which is an estimation result of the attribute of the image 50 of the unsupervised learning data 44, based on an identification target area 62 in the image 50 of the unsupervised learning data 44 that corresponds to the type of attribute to be identified by the first learning model 30.

[0138] Fig. 8 is an explanatory diagram showing an example of the flow of the pseudo label estimation process of this embodiment. An image 50A shown in Fig. 8 is the same as the image 50A shown in Fig. 3A. An image 50D is an example of the image 50.

[0139] The pseudo label estimation unit 22B executes a process of estimating a pseudo label 54 using the image 50 included in the unsupervised learning data 44 acquired by the acquisition unit 20A (step S10).

[0140] Similar to the pseudo label estimation unit 20B, the pseudo label estimation unit 22B determines whether it is difficult to estimate an attribute using the first classification target region 62A in the image 50 of the unsupervised learning data 44. In this embodiment, the pseudo label estimation unit 22B determines whether it is difficult to estimate the gender of the subject S, which is an attribute, using the first classification target region 62A, which is a face image region in the image 50.

[0141] Fig. 8 shows image 50D as an example of image 50 for which it is difficult to estimate attributes using first classification target region 62A, and Fig. 8 also shows image 50A as an example of image 50 for which it is possible to estimate attributes using first classification target region 62A.

[0142] For example, assume that the image 50 included in the unsupervised learning data 44 acquired by the acquisition unit 20A is image 50A (step S12). In image 50A, a first classification target region 62A, which is a facial image region, contains the head of subject S in a state in which gender can be estimated from the first classification target region 62A. Specifically, head features such as eyes, nose, and mouth, which are used to estimate gender, are identifiably captured in the first classification target region 62A of image 50A. In this case, the pseudo label estimation unit 22B can estimate a pseudo label 54, which is an estimation result of gender, from the facial image region, which is the first classification target region 62A of image 50A.

[0143] On the other hand, assume that the image 50 included in the unsupervised learning data 44 acquired by the acquisition unit 20A is image 50D (step S13). In image 50D, the area occupied by subject S is smaller than that in image 50A, and the size of the facial image area of ​​subject S is also small. Specifically, the first classification target area 62A in image 50D has a small facial image area, and head features such as the eyes, nose, and mouth used to estimate gender are captured in an unidentifiable state. In this case, it becomes difficult for the pseudo label estimation unit 22B to estimate a pseudo label 54, which is an estimation result of gender, from the facial image area that is the first classification target area 62A in image 50D.

[0144] Therefore, the pseudo label estimation unit 22B determines whether the state of the subject S represented by the classification target region 62 in the image 50 of the unsupervised learning data 44 satisfies predetermined estimable conditions. As described in the above embodiment, the state of the subject S represented by the classification target region 62 and the estimable conditions may be determined in advance according to the type of attribute of the object to be classified by the first learning model 30.

[0145] As described above, in this embodiment, it is assumed that the first classification target area 62A is the face image area of ​​the subject S, and the type of attribute to be classified by the first learning model 30 is the gender of the subject S.

[0146] In this case, the pseudo label estimation unit 22B uses, for example, the face size of the subject S as the state of the subject S represented by the classification target region 62. The face size is the size of the face image region of the subject S in the image 50. The size of the face image region is represented, for example, by the number of pixels occupied by the face image region in the image 50, the area, the ratio of the number of pixels to the entire image 50, the ratio of the area to the entire image 50, etc.

[0147] Furthermore, the pseudo label estimation unit 22B uses a predetermined threshold value for the face size of the subject S as an estimation possible condition. This threshold value may be determined in advance. For example, this threshold value may be determined in advance as a threshold value for distinguishing between a face size in which the gender can be estimated from the face image area and a face size in which the gender cannot be estimated from the face image area.

[0148] Then, when the face size of subject S included in image 50 is less than the threshold, pseudo label estimation unit 22B determines that the state of subject S represented by classification target region 62 in image 50 does not satisfy the estimation possible conditions, and that it is difficult to estimate attributes using first classification target region 62A in image 50. On the other hand, when the face size of subject S included in image 50 is equal to or greater than the threshold, pseudo label estimation unit 22B determines that the state of subject S represented by classification target region 62 in image 50 satisfies the estimation possible conditions, and that it is possible to estimate attributes using first classification target region 62A in image 50.

[0149] Then, if the pseudo label estimation unit 22B determines that it is difficult to estimate attributes using the first classification target region 62A in the image 50 of the unsupervised learning data 44 (S13), it estimates a pseudo label 54B based on the second classification target region 62B, which is a whole-body region (step S14).

[0150] For example, the pseudo label estimation unit 22B estimates a pseudo label 54B from a second classification target region 62B of an image 50D of the unsupervised learning data 44 using a second learning model 34 that has been trained in advance.

[0151] The second learning model 34, like the second learning model 32 in the above embodiment, is a learning model with a slower processing speed than the first learning model 30. Also, like the second learning model 32 in the above embodiment, the second learning model 34 is a learning model with a larger size than the first learning model 30. Therefore, the second learning model 34 is a model that has a slower processing speed than the first learning model 30 but can output highly accurate classification results.

[0152] The pseudo label estimation unit 22B identifies a whole-body region, which is a second classification target region 62B, from an image 50D included in the unsupervised learning data 44. Then, the pseudo label estimation unit 22B inputs the identified whole-body region, which is a second classification target region 62B, to the second learning model 34, and obtains an attribute, which is gender, as an output from the second learning model 34. Then, by obtaining the attribute output from the second learning model 32, the pseudo label estimation unit 22B estimates the attribute as a pseudo label 54B.

[0153] Then, pseudo label estimation unit 22B generates first supervised training data 42A consisting of pairs of images 50 of unsupervised training data 44 and estimated pseudo labels 54B (step S16).

[0154] On the other hand, if the pseudo label estimation unit 23B determines that it is possible to estimate attributes using the first classification target region 62A in the image 50 of the unsupervised learning data 44 (step S12), it estimates a pseudo label 54A based on the first classification target region 62A (step S15).

[0155] For example, the pseudo label estimation unit 22B estimates a pseudo label 54A from a first classification target region 62A of an image 50A of the unsupervised learning data 44 using the first learning model 30 to be learned.

[0156] The pseudo label estimation unit 22B identifies a facial image region that is a first classification target region 62A from the image 50A included in the unsupervised learning data 44. Then, the pseudo label estimation unit 22B inputs the identified facial image region that is the first classification target region 62A to the first learning model 30, and acquires an attribute that is gender as an output from the first learning model 30. Then, by acquiring the attribute output from the first learning model 30, the pseudo label estimation unit 22B estimates the attribute as a pseudo label 54A.

[0157] Then, the pseudo label estimation unit 22B generates the first supervised training data 42A, which is made up of pairs of the images 50 of the unsupervised training data 44 and the estimated pseudo labels 54A (step S16).

[0158] The learning unit 20C is similar to the learning unit 20C in the above embodiment, except that it uses first supervised training data 42A generated by a pseudo label estimation unit 22B instead of the pseudo label estimation unit 20B.

[0159] Next, an example of the flow of information processing executed by the image processing unit 10B of this embodiment will be described.

[0160] FIG. 9 is a flowchart showing an example of the flow of information processing executed by the image processing unit 10B of this embodiment.

[0161] Acquiring unit 20A acquires training data 40 including second supervised training data 42B and unsupervised training data 44 (step S200).

[0162] The pseudo label estimation unit 22B determines whether the training data 40 to be processed, among the training data 40 acquired by the acquisition unit 20A, is the second supervised training data 42B to which the correct label 52 has been assigned (step S202).

[0163] If the training data 40 to be processed is the second supervised training data 42B to which the correct label 52 has been assigned (step S202: Yes), the pseudo label estimation unit 22B outputs the second supervised training data 42B to the training unit 20C, and the process proceeds to step S218, which will be described later.

[0164] On the other hand, if the training data 40 to be processed is unsupervised training data 44 to which no correct label 52 has been assigned (step S202: No), the process proceeds to step S204.

[0165] In step S204, the pseudo label estimation unit 20B identifies the first classification target region 62A, which is a face image region of the image 50 included in the unsupervised learning data 44 (step S204).

[0166] The pseudo label estimation unit 22B determines whether the face size identified from the face image region of the subject S identified in step S204 is equal to or greater than a threshold, which is an estimation possible condition (step S206). That is, through the processes of steps S204 to S206, the pseudo label estimation unit 22B determines whether the state of the subject S represented by the classification target region 62 of the image 50 included in the unsupervised learning data 44 satisfies the estimation possible condition for estimating an attribute from the first classification target region 62A.

[0167] If the face size is equal to or larger than the threshold (step S206: Yes), the pseudo label estimation unit 22B determines that it is possible to estimate the gender using the first discrimination target region 62A, which is a face image region of the image 50. Then, the process proceeds to step S208.

[0168] In step S208, the pseudo label estimation unit 22B estimates a pseudo label 54A from the first classification target region 62A and the first learning model 30 (step S208). The pseudo label estimation unit 22B inputs the face image region that is the first classification target region 62A included in the image 50 of the unsupervised learning data 44 to the first learning model 30. Then, the pseudo label estimation unit 22B acquires an attribute representing gender as an output from the first learning model 30. By acquiring the attribute output from the first learning model 30, the pseudo label estimation unit 22B estimates the attribute as a pseudo label 54A.

[0169] Then, pseudo label estimation unit 22B generates first supervised training data 42A consisting of pairs of images 50 of unsupervised training data 44 and pseudo labels 54A estimated in step S208 (step S212). Then, the process proceeds to step S218, which will be described later.

[0170] On the other hand, if it is determined in step S206 that the face size is less than the threshold (step S206: No), the pseudo label estimation unit 22B determines that it is difficult to estimate gender using the first classification target region 62A, which is a face image region in the image 50. In other words, if the face size of the subject S is less than the threshold, the pseudo label estimation unit 22B determines that the state of the subject S represented by the classification target region 62 in the image 50 does not satisfy the estimation possible conditions, and that it is difficult to estimate attributes using the first classification target region 62A in the image 50. Then, the process proceeds to step S214.

[0171] In step S214, the pseudo label estimation unit 22B estimates a pseudo label 54B from the second classification target region 62B and the second learning model 32 (step S214). The pseudo label estimation unit 22B inputs the whole-body region, which is the second classification target region 62B included in the image 50 of the unsupervised learning data 44, to the second learning model 32. Then, the pseudo label estimation unit 22B acquires an attribute representing gender as an output from the second learning model 32. By acquiring the attribute output from the second learning model 32, the pseudo label estimation unit 22B estimates the attribute as a pseudo label 54B.

[0172] Then, pseudo label estimation unit 22B generates first supervised training data 42A consisting of pairs of images 50 of unsupervised training data 44 and pseudo labels 54B estimated in step S214 (step S216). Then, the process proceeds to step S218.

[0173] In step S218, the learning unit 20C learns the first learning model 30 using the first recognition target region 62A included in the learning data 40 (step S218).

[0174] The learning unit 20C accepts the second supervised learning data 42B determined in step S202 (step S202: Yes), the first supervised learning data 42A generated in step S212, and the first supervised learning data 42A generated in step S216 as learning data 40. Then, the learning unit 20C identifies a first classification target region 62A, which is a facial image region, from an image 50 included in the learning data 40, and inputs the first classification target region 62A to the first learning model 30. Then, the learning unit 20C acquires the attribute 56, which is gender, output from the first learning model 30 in response to the input of the first classification target region 62A as the attribute 56 estimated by the first learning model 30.

[0175] The output control unit 20D outputs the first learning model 30 learned in step S218 (step S220), and then ends this routine.

[0176] As described above, the pseudo label estimation unit 22B of the image processing device 1B of this embodiment, like the pseudo label estimation unit 20B of the above embodiment, estimates pseudo labels 54 based on the classification target regions 62 in the images 50 of the unsupervised learning data 44 according to the types of attributes to be classified by the first learning model 30 being trained. The learning unit 20C uses the first supervised learning data 42A in which the pseudo labels 54 are assigned to the images 50 of the unsupervised learning data 44 to train the first learning model 30 that classifies the attributes 56 of the images 50.

[0177] Therefore, the image processing device 1B of this embodiment can provide the first learning model 30 (learning model) that can identify the attributes of the image 50 with high accuracy, similar to the image processing device 1 of the above embodiment.

[0178] In other words, the image processing device 1B of this embodiment can provide a first learning model 30 that can identify attributes with high accuracy for a first learning model 30 that targets attributes of a different type than those of the image processing device 1 of the above embodiment.

[0179] It is preferable that the image 50 included in at least one of the unsupervised learning data 44, the first supervised learning data 42A, and the second supervised learning data 42B used in the first and second embodiments be the same type of image as the input image to be processed by the first learning model 30. The input image to be processed by the first learning model 30 is an image used as an input to the first learning model 30 in an information processing device to which the first learning model 30 is applied.

[0180] The same type of images 50 means that the properties of the elements included in the images 50 are the same between the images 50 and the input images. In more detail, the same type of images 50 means that at least one element of the shooting environment, composition conditions, processing conditions, and creation conditions is the same.

[0181] For example, assume that the input image to be input to the first learning model 30 at the application target is a synthetic image. In this case, it is preferable that the image 50 included in at least one of the unsupervised learning data 44, the first supervised learning data 42A, and the second supervised learning data 42B is a synthetic image.

[0182] In addition, assume that the input images input to the first learning model 30 at the application target are images captured in a specific shooting environment. In this case, it is preferable that the images 50 included in at least one of the unsupervised learning data 44, the first supervised learning data 42A, and the second supervised learning data 42B are images captured in the same specific shooting environment.

[0183] By using an image of the same type as the input image as the image 50, the deviation of the classification environment is reduced, and the classification accuracy of the first learning model 30 can be further improved.

[0184] Next, an example of the hardware configuration of the image processing device 1 and the image processing device 1B of the above embodiment will be described.

[0185] FIG. 10 is a diagram showing an example of the hardware configuration of the image processing device 1 and the image processing device 1B according to the above embodiment.

[0186] The image processing device 1 and the image processing device 1B of the above embodiment are configured with a hardware using a typical computer, including a control device such as a CPU (Central Processing Unit) 90D, storage devices such as a ROM (Read Only Memory) 90E, a RAM (Random Access Memory) 90F, and a HDD (Hard Disk Drive) 90G, an I / F unit 90B that interfaces with various devices, an output unit 90A that outputs various information, an input unit 90C that accepts user operations, and a bus 90H that connects each unit. In this case, the control unit 20 in FIG. 1 corresponds to the control device such as the CPU 90D.

[0187] In the image processing device 1 and image processing device 1B of the above embodiment, the CPU 90D reads out a program from the ROM 90E onto the RAM 90F and executes it, thereby realizing the above-mentioned respective units on the computer.

[0188] The programs for executing the above processes executed by the image processing device 1 and image processing device 1B of the above embodiment may be stored in the HDD 90G. Also, the programs for executing the above processes executed by the image processing device 1 and image processing device 1B of the above embodiment may be provided by being pre-installed in the ROM 90E.

[0189] The programs for executing the above processes executed by the image processing device 1 and image processing device 1B of the above embodiments may be stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disc), or flexible disk (FD) and provided as a computer program product. The programs for executing the above processes executed by the image processing device 1 and image processing device 1B of the above embodiments may be stored on a computer connected to a network such as the Internet and provided by downloading via the network. The programs for executing the above processes executed by the image processing device 1 and image processing device 1B of the above embodiments may be provided or distributed via a network such as the Internet.

[0190] In the above description, the image processing device 1 is configured from the image processing unit 10, UI unit 14, and communication unit 16. However, the image processing device according to the present invention may be configured solely with the image processing unit 10. While the above describes an embodiment of the present invention, the embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit and scope of the invention. This embodiment and its modifications are within the scope and spirit of the invention, and are included in the scope of the invention and its equivalents as set forth in the claims. [Explanation of symbols]

[0191] 1, 1B Image processing device 20A Acquisition Department 20B, 22B Pseudo label estimation unit 20C Learning Department 20D Output control section

Claims

1. an acquisition unit that acquires unsupervised learning data consisting of images to which no correct attribute labels have been assigned; a pseudo label estimation unit that estimates pseudo labels, which are estimation results of the attributes of the image of the unsupervised learning data, based on a classification target region in the image of the unsupervised learning data that corresponds to the type of the attribute of the classification target by a first learning model of a learning target; a learning unit that uses first supervised learning data in which the pseudo-label is assigned to the image in the unsupervised learning data to learn the first learning model that identifies the attribute of the image; Equipped with The pseudo label estimation unit When it is determined that it is difficult to estimate the attribute in the image of the unsupervised learning data using a first discrimination target region, which is the discrimination target region used for training the first learning model, estimating the pseudo label based on a second identification target region that is the identification target region different from the first identification target region; Image processing device.

2. The pseudo label estimation unit If it is determined that the attribute can be estimated using the first classification target region in the image of the unsupervised learning data, estimating the pseudo label based on the first identified target region; The image processing device according to claim 1 .

3. The pseudo label estimation unit if the state of the subject represented by the classification target region in the image of the unsupervised learning data does not satisfy a predetermined estimation condition for estimating the attribute from the first classification target region, it is determined that it is difficult to estimate the attribute using the first classification target region; 3. The image processing device according to claim 1.

4. The pseudo label estimation unit when it is determined that it is difficult to estimate the attribute using the first classification target region in the image of the unsupervised learning data, estimating the pseudo label that is predetermined depending on the state of the subject represented by the second classification target region; The image processing device according to claim 1 .

5. The pseudo label estimation unit If it is determined that the attribute can be estimated using the first identification target region, the pseudo label is estimated from the first identification target region of the image of the unsupervised learning data using a second learning model that has been trained in advance. The image processing device according to claim 2 .

6. The pseudo label estimation unit If it is determined that it is difficult to estimate the attribute using the first classification target region in the image of the unsupervised learning data, using a pre-trained second learning model, estimating the pseudo label from the second discrimination target region of the image of the unsupervised learning data; The image processing device according to claim 1 .

7. The pseudo label estimation unit If it is determined that the attribute can be estimated using the first classification target region in the image of the unsupervised learning data, using the first learning model to estimate the pseudo label from the first target region of the image of the unsupervised learning data; The image processing device according to claim 2 .

8. The first learning model is a learning model having a faster processing speed than the second learning model.

7. The image processing device according to claim 5 or 6.

9. The acquisition unit Further, second supervised training data is acquired, which is made up of the images to which the correct labels have been assigned; The learning unit training the first learning model using the first supervised learning data and the second supervised learning data; The image processing device according to claim 1 .

10. the image included in at least one of the unsupervised learning data, the first supervised learning data, and the second supervised learning data is an image of the same type as an input image to be processed by the first learning model; The image processing device according to claim 9 .

11. Executed by the control unit, An acquisition step of acquiring unsupervised learning data consisting of images to which no correct attribute labels have been assigned; a pseudo label estimation step of estimating pseudo labels, which are estimation results of the attributes of the image of the unsupervised learning data, based on a discrimination target region in the image of the unsupervised learning data that corresponds to the type of the attribute of the discrimination target by a first learning model of a learning target; a learning step of learning the first learning model that identifies the attributes of the images using first supervised learning data in which the pseudo-labels are assigned to the images of the unsupervised learning data; Including, The pseudo label estimation step includes: When it is determined that it is difficult to estimate the attribute in the image of the unsupervised learning data using a first discrimination target region, which is the discrimination target region used for training the first learning model, estimating the pseudo label based on a second identification target region that is the identification target region different from the first identification target region; Image processing methods.

12. An image processing program to be executed by a computer, An acquisition step of acquiring unsupervised learning data consisting of images to which no correct attribute labels have been assigned; a pseudo label estimation step of estimating pseudo labels, which are estimation results of the attributes of the image of the unsupervised learning data, based on a discrimination target region in the image of the unsupervised learning data that corresponds to the type of the attribute of the discrimination target by a first learning model of a learning target; a learning step of learning the first learning model that identifies the attributes of the images using first supervised learning data in which the pseudo-labels are assigned to the images of the unsupervised learning data; Including, The pseudo label estimation step includes: When it is determined that it is difficult to estimate the attribute in the image of the unsupervised learning data using a first discrimination target region, which is the discrimination target region used for training the first learning model, estimating the pseudo label based on a second identification target region that is the identification target region different from the first identification target region; Image processing program.

Images