Image recognition support device, image recognition support method, and image recognition support program

Abstract

To provide an image recognition support apparatus for supporting generation of a model for recognizing an attribute in an image with high accuracy, an image recognition support method, and an image recognition support program.SOLUTION: An image recognition support apparatus 10 includes: an image input unit 201 which acquires an image; a pseudo-label generation unit 202 which recognizes the acquired image on the basis of each of multiple kinds of image recognition models to output recognition information, and generates a pseudo-label indicating an attribute of the acquired image, on the basis of each piece of output recognition information; and a new label generation unit 203 which generates a new label on the basis of the generated pseudo-label.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image recognition support device, an image recognition support method, and an image recognition support program.

Background Art

[0002] By automatically recognizing a photographed image, it is possible to identify the attributes of each subject in the image and know the events recorded in the image. For example, aerial images and satellite images of a disaster site are useful as means for remotely grasping the situation on the ground. In particular, by simultaneously recognizing a plurality of attributes in a wide-area image, it becomes possible to quickly grasp the disaster situation. For such a purpose, in order to create a classifier that automatically recognizes and classifies each attribute photographed in an image, it is necessary to prepare a pair of an image and a label (correct label) indicating all the attributes in the image as learning data, and let the classifier learn these pairs as patterns. Here, a pair of an image and a label is called learning data.

[0003] However, in preparing the learning data, it is difficult to accurately prepare all the correct labels in a wide-area image. In particular, when a person sets a label for an image, there is a problem that an incorrect label is given to an attribute that is not correct, or a missing label is generated in which the label is not given even though the attribute exists. Further, when learning a pattern of an image and a label, there is a problem that if a classifier is created using learning data including an incorrect label, the accuracy of the classifier decreases. Therefore, in order to correctly recognize the attributes in a target image without reducing the accuracy of the classifier, it is necessary to correct the labels attached to the image by a person.

[0004] As an image recognition support device and method for correcting labels set by a person, for example, Patent Document 1 is known. Patent Document 1 describes that the reliability regarding an attribute output from an image recognition unit is acquired, and the acquired reliability is compared with preset label information in a display unit to correct the label.

Prior Art Documents

Patent Document

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In the technology of Patent Document 1, it is necessary for the user to manually correct the label by comparing the preset label information with the reliability output from the image recognition unit. However, it takes a very long time to manually correct all of a large number of images. On the other hand, when automatically correcting the label based on the reliability output from the image recognition unit, if the accuracy of the output reliability is not sufficiently high, the pattern will be learned based on the incorrect label information, which will also lead to a decrease in the accuracy of the image recognition unit.

[0007] Therefore, an object of the present invention is to provide an image recognition support device, an image recognition support method, and an image recognition support program capable of assisting in creating a model for accurately recognizing attributes in an image.

Means for Solving the Problems

[0008] One aspect of the present invention for solving the above problems is an image recognition support device including an image input unit that acquires an image, Calculate the occurrence frequency of each attribute of the input image based on a predetermined image database, and input the obtained image into each of a plurality of classifiers, which are image recognition models with different tendencies of recognition information of the attributes of the input image output according to the occurrence frequency of the attributes of the input image. Generate integrated recognition information for each attribute of the obtained image based on the recognition information of each attribute of the obtained image output from each of the plurality of classifiers, and generate pseudo-labels based on the integrated recognition information. a pseudo-label generation unit, Calculate the accuracy of each attribute of the pseudo-label, and based on the calculated accuracies, for each attribute of the obtained image and a new label generation unit that generates a new label.

[0009] Also, one aspect of the present invention for solving the above problems is an image recognition support method in which an information processing device executes an image input process for acquiring an image, Calculate the occurrence frequency of each attribute of the input image based on a predetermined image database, and input the obtained image into each of a plurality of classifiers, which are image recognition models with different tendencies of recognition information of the attributes of the input image output according to the occurrence frequency of the attributes of the input image. Generate integrated recognition information for each attribute of the obtained image based on the recognition information of each attribute of the obtained image output from each of the plurality of classifiers, and generate pseudo-labels based on the integrated recognition information. a pseudo-label generation process, Calculate the accuracy of each attribute of the pseudo-label, and based on the calculated accuracies, for each attribute of the obtained image and a new label generation process for generating a new label.

[0010] Another aspect of the present invention for solving the above problems is to cause an information processing apparatus to execute an image input process for acquiring an image, Calculate the occurrence frequency of each attribute of the input image based on a predetermined image database, and input the obtained image into each of a plurality of classifiers, which are image recognition models with different tendencies of recognition information of the attributes of the input image output according to the occurrence frequency of the attributes of the input image. Generate integrated recognition information for each attribute of the obtained image based on the recognition information of each attribute of the obtained image output from each of the plurality of classifiers, and generate pseudo-labels based on the integrated recognition information. a pseudo-label generation process, Calculate the accuracy of each attribute of the pseudo-label, and based on the calculated accuracies, for each attribute of the obtained image and a new label generation process for generating a new label, and is an image recognition support program.

Advantages of the Invention

[0011] According to the present invention, since the new label generation unit (process) generates a new label based on the pseudo-labels generated by the pseudo-label generation unit (process) based on a plurality of types of image recognition models, it is possible to gradually generate new labels with higher reliability from the pseudo-labels obtained in the middle. As a result, labels of highly reliable learning data can be generated without manual confirmation (visual inspection, etc.). And thereby, it becomes possible to generate a highly accurate learning model from data including errors. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. For the sake of clarity of the description, the following description and drawings are appropriately omitted and simplified. Also, the present invention is not limited to this embodiment, and all application examples that conform to the idea of the present invention are included in the technical scope of the present invention. Also, unless otherwise specified, the number of each component mentioned may be singular or plural.

[0014] <System Configuration> FIG. 1 is a diagram for explaining the outline of the configuration of the image recognition system 1 according to this embodiment. The image recognition system 1 includes a photographing system 101 that photographs an image, and an image recognition support device 10 that generates an image recognition model based on the image photographed by the photographing system 101. Between the photographing system 101 and the image recognition support device 10, for example, a wired or wireless network 5 such as a LAN (Local Area Network), WAN (Wide Area Network), the Internet, or a dedicated line is communicably connected.

[0015] The imaging system 101 is configured to include one or more imaging devices (cameras) that capture images. The imaging device may be, for example, an imaging device held by a person, a device fixed to the ground surface, an imaging device provided on a vehicle moving on the ground surface, or an imaging device provided on a drone or an aircraft.

[0016] The images captured by the imaging system 101 include images of one or more subjects (objects), and the user can associate each object with any one of a plurality of pre-listed attributes (categories). The attributes may be, for example, a person, an artifact such as a building, a vehicle, or a road, a natural object such as a sea or a river, or something indicating a state of an object or a person such as a flood, a building collapse, a traffic jam, or a crowded state of people.

[0017] Also, each image may be either a color image or a monochrome image. Further, each image may be, in addition to an image captured by a camera, a SAR image (SAR: Synthetic Aperture Radar), a CG image (CG: Computer Graphics), or any other type of image that has been acquired in advance. Also each image may be accompanied by meta information.

[0018] The image recognition support device 10 acquires the images captured by the imaging system 101. The image recognition support device 10 generates a plurality of classifiers (trained models) having different configurations with respect to the recognition of image attributes. Then, the image recognition support device 10 inputs the image (designated image) specified by the user to each generated classifier, and compares the value of each label (pseudo label) obtained from the output value of each classifier with the value of the label (original label) preset by the user for the designated image for each attribute, thereby generating a new label (new label).

[0019] The new label generated in this way is information that correctly reflects the attributes of the designated image.

[0020] Thereafter, the image recognition assistance device 10 generates an image recognition model by performing machine learning to learn the relationship between a plurality of combinations of the specified images and the new labels.

[0021] <Original label, pseudo label, and new label> 2 is a diagram showing an example of an original label, a pseudo label, and a new label, As shown in the figure, label information 402 (original label, pseudo label, and new label) is set for an image 401.

[0022] The label information 402 is information on the probability that the image 401 has a certain attribute at a learning time point (epoch) in the machine learning process of the classifier described later, or information indicating whether a certain attribute exists in the image 401. The original label 402a of the label information 402 is set in advance by a user or the like, but may contain an error. The pseudo label 402b is automatically set by the pseudo label generation unit 202 described later. The new label 402c is automatically set by the new label generation unit 203 described later based on the original label 402a and the pseudo label 402b.

[0023] Regarding the notation of the label information 402 in the figure, for example, "x123" means the image with ID "1". "y342" indicates that it is the original label for attribute "3" that was set when the epoch (number of machine learning trials) for the image with ID "3" was "2". "y342" indicates that it is the pseudo label for attribute "2" that was set when the epoch for the image with ID "3" was "4". "z567" indicates that it is the new label for attribute "7" that was set when the epoch for the image with ID "5" was "6". Note that the pseudo label and new label may differ for each epoch, but the original label is common to all epochs.

[0024] Next, FIG. 3 is a diagram showing another example of the original label, pseudo label, and new label for an image. As shown in the figure, label information 404 (original label, pseudo label, and new label) is set for image 403. The original label 404a, pseudo label 404b, and new label 404c of the label information 404 are the same as the original label 402a, pseudo label 402b, and new label 402c described above. And, different from FIG. 2, coordinate information 405 indicating the position of the subject of each attribute in image 403 is added to the label information 404.

[0025] As shown in the above label information 402 and 404, the image recognition support device 10 creates new labels 402c and 404c, which are more accurate labels than the original labels 402a and 404a that may contain errors, and uses these for the image recognition model, so that all attributes included in the image for which the user desires image recognition can be correctly recognized.

[0026] <Image Recognition Support Device> Next, FIG. 4 is a block diagram for explaining an example of the functions provided in the image recognition support device 10. The image recognition support device 10 includes functional units (programs) such as an image input unit 201, a pseudo label generation unit 202, a new label generation unit 203, a classifier storage unit 204, and an image recognition model generation unit 206. Also, the image recognition support device 10 stores an integrated DB 205 (DB: Database).

[0027] The image input unit 201 acquires the image captured by the imaging system 101 and stores the acquired image in the integrated DB 205. Also, the image input unit 201 inputs each image to the pseudo label generation unit 202.

[0028] The pseudo label generation unit 202 recognizes the image acquired by the image input unit 201 based on each of a plurality of types of image recognition models (classifiers), outputs recognition information, and generates a pseudo label indicating the attributes of the acquired image based on each of the output recognition information.

[0029] Specifically, first, the pseudo-label generation unit 202 generates and stores a plurality of types of classifiers that take an image as input and output recognition information for each attribute of the image. In this embodiment, the recognition information is the reliability, which is the probability (likelihood) that the image has the attribute.

[0030] Each classifier is generated based on each image stored in the integrated DB 205 and its label (original label, and if it exists, a new label to be described later), and is generated such that the tendency of the recognition information of each attribute output by the characteristic value related to each attribute of the image is different.

[0031] Note that in this embodiment, the characteristic value of each attribute is the appearance frequency of each attribute (the probability that the attribute exists in the image). The appearance frequency here may be the appearance frequency of each attribute in all the images captured by the imaging system 101 so far, the appearance frequency of each attribute in a specific image group, or other statistically derived appearance frequencies.

[0032] Also, each classifier is a learned model generated based on deep learning. As such a classifier, for example, there is a convolutional neural network (CNN) composed of an information network having a plurality of layers.

[0033] Then, the pseudo-label generation unit 202 inputs the specified image input from the image input unit 201 to each of the plurality of classifiers, recognizes each attribute of the image, and outputs the result as recognition information.

[0034] Then, the pseudo-label generation unit 202 calculates the integrated recognition information (hereinafter also referred to as integrated reliability. Details will be described later) of each attribute of the specified image based on the recognition information (reliability) of each attribute output from each of the plurality of classifiers and a predetermined coefficient (weight coefficient) associated with each combination of each classifier and each attribute, and generates a pseudo-label obtained by performing a predetermined conversion on this. ​​ The pseudo-label generation unit 202 inputs the integrated recognition information and pseudo-labels of each attribute to the new label generation unit 203.

[0035] The new label generation unit 203 generates new labels based on the pseudo-labels generated by the pseudo-label generation unit 202.

[0036] Specifically, the new label generation unit 203 calculates the accuracy (label correct rate) for each attribute of the pseudo-label based on the integrated recognition information of each attribute input from the pseudo-label generation unit 202. Based on the calculated accuracies, the new label generation unit 203 Specified image generates new labels for each attribute of the specified image by correcting the pseudo-labels of each attribute.

[0037] Note that the generated new labels are stored in the integrated DB 205. Also, the generated new labels are repeatedly used in the machine learning of each classifier performed by the pseudo-label generation unit 202.

[0038] The classifier storage unit 204 stores each classifier. Also, the classifier storage unit 204 stores information on the accuracy of pseudo-labels, learning parameters in each classifier, and recognition information of each attribute by each classifier, etc. Note that these information are used, for example, when generating pseudo-labels by a classifier or during the machine learning of a classifier.

[0039] The integrated DB 205 stores the original labels, label information, shooting time, and map information, etc. of each image. For example, the integrated DB 205 stores the ID, epoch, original label, pseudo-label, and new label of each image.

[0040] The image recognition model generation unit 206 generates a learned model (image recognition model) for performing attribute recognition of images based on each image stored in the integrated DB 205 and the new labels of those images. For example, the image recognition model generation unit 206 learns the relationship between a plurality of specified images and the new labels corresponding to each specified image, thereby generating a learned model that takes an image as an input value and outputs recognition information (such as confidence) of each attribute possessed by the image. Note that this learned model is configured as, for example, a neural network (Convolutional Neural Network) having a plurality of layers.

[0041] (Integrated DB) Here, FIG. 5 is a diagram showing an example of information stored in the integrated DB 205. The integrated DB 205 has data items of an ID 302 for which an identifier of each image is set, a shooting time 303 for which the shooting date and time of each image are set, an epoch 304 for which an epoch (specifically, the number of trials in which the processes of S1002 to S1007 described later are executed) is set, an original label 305 for which the original label set for each attribute of each image is set, a pseudo label 306 for which the pseudo label set by a classifier for each attribute of each image is set, and a new label 307.

[0042] Note that recognition information (such as confidence) of a plurality of attributes included in the image is set for the original label 305, the pseudo label 306, and the new label 307. Also, it is assumed that the user has set the original label 305 in advance. Also, the data items described here are examples, and for example, information such as image metadata may be included.

[0043] (Pseudo Label Generation Unit) Next, FIG. 6 is a diagram for explaining the details of the pseudo label generation unit 202. The pseudo label generation unit 202 includes functional units (programs) of an ensemble target selection unit 601, an attribute weight estimation unit 602, an attribute score ensemble processing unit 603, and a pseudo label generation unit 604.

[0044] The ensemble target selection unit 601 selects a plurality of classifiers to be used for generating pseudo labels.

[0045] Based on the characteristic values (appearance frequencies in this embodiment) of each attribute of the images in the integrated DB 205, the attribute weight degree estimation unit 602 sets values (weight coefficients) regarding the learning weights of each attribute in the machine learning of each classifier. In this embodiment, it is assumed that the weight coefficients are automatically calculated based on the characteristic values of each attribute and the hyperparameters of each attribute in the classifier.

[0046] Based on the output values (confidence levels for each attribute) of the images input to each classifier selected by the ensemble target selection unit 601 and the weight coefficients set by the attribute weight degree estimation unit 602, the attribute score ensemble processing unit 603 calculates recognition information (integrated confidence level) for each attribute of the image.

[0047] The pseudo label generation unit 604 converts the integrated confidence levels of each attribute calculated by the attribute score ensemble processing unit 603 into pseudo labels. For example, the pseudo label generation unit 604 converts the integrated confidence level, which is a continuous value, into a value of the pseudo label, which is a discrete value (e.g., 0 or 1).

[0048] (New label generation unit) Next, FIG. 7 is a diagram for explaining the details of the new label generation unit 203. The new label generation unit 203 includes each functional unit (program) of a pseudo label processing unit 801, an attribute threshold setting unit 802, a label fusion unit 803, and a new label conversion unit 804.

[0049] The pseudo label processing unit 801 performs the same processing (conversion from confidence level to label value) as the pseudo label generation unit 604. The pseudo label processing unit 801 performs this processing when the above processing has not been executed in the pseudo label generation process.

[0050] The attribute threshold setting unit 802 sets a parameter (threshold) for determining whether to use the pseudo label as the new label of the specified image by comparing the pseudo label generated by the pseudo label generation unit 202 with the original label for each attribute of the specified image. Specifically, when the recognition accuracy of a certain attribute (the probability that the value of the original label related to a certain attribute is the same as the value of the pseudo label) is high, the attribute threshold setting unit 802 sets a high value for the threshold related to that attribute, and when the recognition accuracy of a certain attribute is low, the attribute threshold setting unit 802 sets a low value for the threshold related to that attribute.

[0051] The new label fusion unit 803 generates a new label for each attribute based on the pseudo label for each attribute generated by the pseudo label processing unit 801 and the threshold for each attribute generated by the attribute threshold setting unit 802.

[0052] The new label conversion unit 804 performs label conversion in the same manner as the pseudo label generation unit 604. For example, in the new label conversion unit 804, the value of the new label of each attribute generated by the new label fusion unit 803 is the confidence level. When the confidence level of a certain attribute is 0.5 or more, "1", which means that the image has that attribute, is set as the value of the new label. When the confidence level of a certain attribute is less than 0.5, "0", which means that the image does not have that attribute, is set as the value of the new label. Note that the value conversion method described here is an example, and any other arbitrary method can be adopted. Here, FIG. 8 is a diagram showing an example of the hardware included in the image recognition support device 10. The image recognition support device 10 includes a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), etc. for processing

[0053] Here, FIG. 8 is a diagram showing an example of the hardware included in the image recognition support device 10. The image recognition support device 10 includes a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), etc. for processing The apparatus 103 is provided with a storage device 104 composed of a memory device or a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), and an SSD (Solid State Drive), a display device 105 composed of a liquid crystal display or an organic EL (Electro-Luminescence) display, etc., an input device 106 composed of a mouse, a keyboard, etc., and a communication device 102 composed of a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, or a serial communication module, etc.

[0054] Note that the display device 105 displays the image captured by the imaging system 101 and the information of each label (pseudo label, new label, etc.).

[0055] Also, the input device 106 receives an input from the user. For example, the input device 106 receives an input for switching the classifier to be displayed by the display device 105, and also receives from the user the setting or modification of the label related to the image captured by the imaging system 101.

[0056] Each function of the image recognition support apparatus 10 is realized by the processing device 103 reading and executing the program stored in the storage device 104. Also, the above program can be recorded on a recording medium and distributed, for example. Next, the processing performed by the image recognition support apparatus 10 will be described.

[0057] <Image Recognition Support Processing> FIG. 9 is a flowchart for explaining an example of the image recognition support processing performed by the image recognition support apparatus 10. This processing is started, for example, when the image captured by the imaging system 101 is stored in the integrated DB 205, or when a predetermined input is given from the user to the image recognition support apparatus 10.

[0058] First, the pseudo-label generation unit 202 calculates the occurrence frequency of each attribute based on the original label of each image stored in the integrated DB 205, and stores the result (step S1001).

[0059] In the present embodiment, the pseudo-label generation unit 202 identifies an attribute that an image has with a frequency less than a predetermined first threshold as a "low-frequency attribute", and the image as a "low-frequency image", respectively, and identifies an attribute that an image has with a frequency equal to or greater than the first threshold and less than a second threshold as a "medium-frequency attribute", and the image as a "medium-frequency image", and identifies an attribute that an image has with a frequency equal to or greater than the second threshold as a "high-frequency attribute", and the image as a "high-frequency image".

[0060] Also, the pseudo-label generation unit 202 generates information on a list of all images stored in the integrated DB 205 as a batch list, and stores this (step S1002). The information in this batch list includes, for example, index information of each image and information on the label (original label or new label) related to the image. Note that the information in this batch list is information generally used in deep learning.

[0061] Then, the pseudo-label generation unit 202 selects one image from the batch list generated in step S1002, and extracts information such as the original label and new label of the selected image (step S1003). When step S1003 is executed for the first time, the information on the new label does not exist. exist.

[0062] The pseudo-label generation unit 202 executes a pseudo-label generation process of creating a pseudo-label based on the image selected in step S1003 and the information on each label thereof (step S1004). Details of the pseudo-label generation process S1004 will be described later.

[0063] In this embodiment, the pseudo-label generation unit 202 generates a pseudo-label by generating a plurality of different classifiers according to the appearance frequency of attributes. That is, the pseudo-label generation unit 202 generates a low-frequency emphasis model that emphasizes learning the low-frequency images among the high-frequency images, medium-frequency images, and low-frequency images, a medium-frequency emphasis model that emphasizes learning the medium-frequency images, and a high-frequency emphasis model that emphasizes learning the high-frequency images. Specifically, the pseudo-label generation unit 202 generates a classifier by maximizing the learning weights of the high-frequency images, medium-frequency images, and low-frequency images respectively.

[0064] In this way, by providing classifiers with a plurality of different characteristics, higher-precision image recognition performance can be realized than when only a single classifier is used.

[0065] The new label generation unit 203 executes a new label generation process S1005 for generating a new label based on the pseudo-labels of each attribute generated by the pseudo-label generation unit 202 and the accuracy (correct rate) of the pseudo-labels of each attribute of each classifier (step S1005). The generated new label is used for the learning of the classifier instead of the original label at the next epoch. The details of the new label generation process S1005 will be described later.

[0066] The new label generation unit 203 checks whether all the images have been selected from the batch list so far (step S1006). If there are images that have not been selected from the batch list (step S1006: No), the pseudo-label generation unit 202 selects one of those images and repeats the processing from step S1003 and later to continue the machine learning of the neural network. If there are no images that have not been selected from the batch list (step S1006: Yes), the processing of step S1007 is performed.

[0067] In step S1007, the new label generation unit 203 checks whether a predetermined number of epochs (number of learning iterations) has been reached. If the number of epochs has not been reached (step S1007: No), the pseudo label generation unit 202 repeats the processes after step S1002. If the number of epochs has been reached (step S1007: Yes), the process of step S1008 is performed.

[0068] In step S1008, the new label generation unit 203 stores in the integrated database 205 the pseudo labels and new labels of each image and the recognition results for each attribute by each classifier.

[0069] Then, the new label generation unit 203 determines whether a predetermined number of repetitions (iterations) has been reached (step S1009). If the number of iterations has not been reached (step S1009: No), the pseudo label generation unit 202 repeats the processes after step S1002. If the number of iterations has been reached (step S1009: Yes), the pseudo label generation unit 202 performs the process of step S1010.

[0070] After that, the image recognition support device 10 generates an image recognition model by performing machine learning to learn the relationship between each image and the combination of new labels stored in the integrated database 205 (step S1010). The user can input an image for which the user wants to perform attribute recognition to this image recognition model to output the attributes of the image. The image recognition support process ends here.

[0071] <Pseudo Label Generation Process> FIG. 10 is a flowchart for explaining the details of the pseudo label generation process S1004.

[0072] The pseudo label generation unit 202 obtains the reliability of each attribute of the specified image output from each classifier by inputting the specified image to each classifier (step S701). Note that the reliability indicates, for example, the probability that each attribute exists in the specified image in the range of 0 to 1.

[0073] The ensemble target selection unit 601 selects a classifier to be used for generating pseudo labels from among all the classifiers stored in the classifier storage unit 204 (step S702).

[0074] In this case, the ensemble target selection unit 601 may select all the classifiers, or may select only classifiers with good recognition results (for example, classifiers that have recognized correct attributes with a predetermined probability or more in previous processing) among the classifiers, or may select classifiers according to other predetermined criteria.

[0075] The attribute weight estimation unit 602 sets the weight coefficient of each attribute in each classifier (at the time of learning) based on the appearance frequency of each attribute obtained in step S1001 (step S703).

[0076] For example, the attribute weight estimation unit 602 sets the weight coefficient of the output for low-frequency images in the high-frequency emphasis model to 0.3 (a low value), and the weight coefficient of the output for low-frequency images in the low-frequency emphasis model to 0.7 (a high value). The weight coefficient is automatically determined, for example, by setting it as a hyperparameter (appearance frequency of attributes) in a neural network.

[0077] The attribute score ensemble processing unit 603 calculates an integrated reliability based on the classifier selected in step S702 and the weight coefficient of each attribute set in step S703 (step S704).

[0078] FIG. 11 is a diagram for explaining an example of a method for calculating reliability. As shown in the figure, there are three classifiers, namely, a low-frequency emphasis model 51, a medium-frequency emphasis model 52, and a high-frequency emphasis model 53. Assume that there are attributes 1 and 2 as low-frequency attributes, which are the attributes most emphasized and learned in the low-frequency emphasis model 51, attributes 3 and 4 as medium-frequency attributes, which are the attributes most emphasized and learned in the medium-frequency emphasis model 52, and attributes 5 and 6 as high-frequency attributes, which are the attributes most emphasized and learned in the high-frequency emphasis model 53.

[0079] First, in step s701, the attribute score ensemble processing unit 603 inputs the specified image to each classifier (low-frequency emphasis model 51, medium-frequency emphasis model 52, and high-frequency emphasis model 53), thereby calculating the recognition results of the specified image by each classifier (reliabilities 54 of low-frequency attributes 1 and 2, reliabilities 55 of medium-frequency attributes 3 and 4, and reliabilities 56 of high-frequency attributes 5 and 6). Then, for each attribute, the attribute score ensemble processing unit 603 multiplies the reliabilities 54, 55, and 56 calculated by each classifier by the weight coefficient 57 set for each classifier and attribute, and sums these multiplication values to obtain the integrated reliability 58.

[0080] Note that since the low-frequency emphasis model can recognize low-frequency attributes with high accuracy compared to other models, the weight coefficient (0.7) for the low-frequency images of the low-frequency model is set to a value larger than the weight coefficients for the low-frequency images of other models (high-frequency model: 0.1, and medium-frequency model: 0.2). The same applies to other attributes. In the medium-frequency emphasis model, high-value weight coefficients are set while emphasizing medium-frequency attributes. Attach importance to model's weight coefficient for low-frequency images (0.7) is set to a value larger than the weight coefficients for low-frequency images of other models (high-frequency Attach importance to model: 0.1, and medium-frequency Attach importance to model: 0.2). The same applies to other attributes. In the medium-frequency emphasis model, high-value weight coefficients are set while emphasizing medium-frequency attributes.

[0081] Here, an example of the method for setting the weight coefficients of each classifier will be described. As the loss function in the machine learning of each classifier Focal Loss (FL(p t )) = -(1 - p t ) γ × log(p t )

[0082] is used, the larger the coefficient γ in the Focal Loss, the more difficult the recognition is, that is, it can be configured to emphasize data of low-frequency attributes. For example, the coefficient γ1 of the low-frequency Attach importance to model is 3.0, the coefficient γ2 of the medium-frequency Attach importance to model is 2.0, the coefficient γ of the high-frequency Attach importance to When the coefficient γ3 of the model is preset to 1.0, the coefficients can be set using values such as the mean value = 0 and the values of a normal distribution with variance σ. Here, the value of the normal distribution is the value of the probability density function corresponding to the input variable x. For example, low frequency Attach importance to The weight coefficient of the model for low-frequency images (0.7 in the example of FIG. 11) is the value of the probability density function when x = 0 in the normal distribution. Medium frequency Attach importance to The weight coefficient of the model for images of low-frequency attributes (0.2 in the example of FIG. 11) is the value of the probability density function when x = |γ1 - γ2| in the normal distribution 。 High frequency Attach importance to The weight coefficient of the model for low-frequency images (0.1 in the example of FIG. 11) is the value of the probability density function when x = |γ1 - γ3| in the normal distribution. By normalizing these values (for example, making the sum of these values equal to 1), each weight coefficient is calculated (in the example of FIG. 11, 0.7 + 0.2 + 0.1 = 1). Thereby, for each classifier, it is possible to automatically set weight coefficients that emphasize attributes for which the calculated reliability tends to be high.

[0083] Next, as shown in FIG. 10, the pseudo-label generation unit 604 generates pseudo-labels (step S705) based on the reliability of each attribute calculated in step S704.

[0084] Specifically, the pseudo-label generation unit 604 converts each reliability into a discrete value. For example, when the value of the reliability of an attribute is 0.5 or more, the value of the pseudo-label of that attribute is set to "1" indicating the presence of that attribute, and when the value of the reliability of an attribute is less than 0.5 the value of the pseudo-label of that attribute is set to "0" indicating the absence of that attribute. Note that the values of the pseudo-labels described here are just examples, and values may be set by any other arbitrary method.

[0085] The pseudo-label generation unit 604 stores the information of the pseudo-labels generated in step S705 in the integrated DB 205 (step S706). Specifically, the pseudo-label generation unit 604 sets the values of the pseudo-labels for each attribute in the pseudo-label 306 of the integrated DB 205.

[0086] As described above, the pseudo-label generation unit 202 generates and stores pseudo-labels by integrating and converting the recognition results for each attribute of each classifier (trained model).

[0087] <New label generation process> FIG. 12 is a flowchart for explaining the details of the new label generation process S1005. The new label generation unit 203 acquires the pseudo-labels generated in the pseudo-label generation process S1004 and inputs the acquired pseudo-labels to the pseudo-label processing unit 801 (step S901).

[0088] Also, the new label generation unit 203 calculates the accuracy rate, which is the recognition result for each attribute (step S902). For example, the new label generation unit 203 calculates the accuracy rate for each attribute of the image by comparing the integrated confidence level calculated by each classifier in the pseudo-label generation process S1004 with the value of the original label for each attribute. Note that the method for calculating the accuracy rate described here is an example, and the new label generation unit 203 may evaluate the accuracy of the pseudo-labels for each attribute of the image by any other method.

[0089] Note that if the conversion in step S705 has not been performed on the values of the pseudo-labels acquired in step S901, the pseudo-label processing unit 801 sets values for the pseudo-labels of each attribute in the same manner as in step S705 (step S903). The attribute threshold setting unit 802 sets thresholds for each attribute based on the accuracy rate calculated in step S902 (step S904).

[0090]

[0091] ​For example, if the recognition accuracy of Attribute 1 in the image is 10%, the pseudo-label accuracy of Attribute 1 is considered low. Therefore, the attribute threshold setting unit 802 sets a threshold such that the ratio of using the pseudo-label becomes 0.1 times that of the original label. If the accuracy of Attribute 1 in the image is 95%, the pseudo-label accuracy is considered high, so the attribute threshold setting unit 802 sets a threshold such that the ratio of using the pseudo-label becomes 1 times that of the original label. The attribute threshold setting unit 802 performs all these settings for all attributes. Note that the attribute threshold setting unit 802 may set the threshold based on an input from the user, or may automatically determine it based on the value of each recognition accuracy.

[0092] The new label fusion unit 803 generates a new label for each attribute (step S905) based on the pseudo-label of each attribute calculated in step S902 (step S903) and the threshold of each attribute set in step S904.

[0093] For example, for Attributes 1 to 5 respectively, the original label is (1, 1, 0, 0, 1), the pseudo-label is (1, 0, 0, 1, 1), and the threshold is (1, 1, 1, 0, 1), and the accuracies of Attributes 1 to 5 are 80%, 70%, 90%, 20%, 95% respectively. Since the recognition accuracy of the pseudo-label of Attribute 4 is low, the new label fusion unit 803 sets the new label of Attribute 4 to the original label (instead of the pseudo-label). In this way, the new label fusion unit 803 calculates the new labels of Attributes 1 to 5 as (1 + 1) / 2 = 1, (1 + 0) / 2 = 0.5, (0 + 0) / 2 = 0, 0, (1 + 1) / 2 = 1) respectively using the thresholds of each attribute.

[0094] The new label conversion unit 804 converts the new label calculated in step S905 (step S906) in the same way as the pseudo-label generation unit 604.

[0095] For example, when the new label conversion unit 804 sets the new label as a discrete value, if the value of the new label calculated in step S905 is 0.5 or more, it sets "1" indicating that the image has the corresponding attribute, and if the value of the new label is less than 0.5, it sets Set "0". The method for converting new labels is not limited to what is described here, and various methods may be used.

[0096] The classifier storage unit 204 stores the new label generated in step S906 in the integrated DB205. The classifier storage unit 204 stores the image, each attribute, the new label, the classifier, and the epoch in association with each other in the integrated DB205.

[0097] As described above, the image recognition support device 10 generates and saves new labels for each attribute of the image by integrating the pseudo-labels for each attribute of each classifier.

[0098] <Operation screen> FIG. 13 is a diagram showing an example of the configuration of an operation screen 150 displayed by the image recognition support device 10. This operation screen 150 includes a recognition target video display column 501, a recognition target map display column 502, a recognition result display column 503, a similar video display column 504, a model switching column 505, and a communication menu 506.

[0099] In the recognition target video display column 501, an image (recognition image) in which each attribute is recognized by the classifier is displayed. Here, the position of the recognized attribute (such as an object) may be displayed above the image in the recognition target video display column 501 or at other predetermined positions.

[0100] In the recognition target map display column 502, information such as the latitude and longitude of the location where the recognition image was acquired, and a map of that area are displayed. The map in this case is not limited to two dimensions, and may be displayed in three dimensions if there is altitude information.

[0101] In the recognition result display column 503, information on each attribute output from the classifier and its related information (information such as the confidence level for each attribute, pseudo label, accuracy rate, new label, etc.) are displayed. Note that in the recognition result display column 503, not all attribute information may be displayed, but only the attribute information according to a certain criterion or specified by the user. For example, only the pseudo label, accuracy rate, and information on the new label of attributes with a confidence level above a certain value may be displayed.

[0102] Note that on the operation screen 150, a new label correction column 507 for receiving input from the user to correct the displayed new label may be provided.

[0103] In the similar video display column 504, other images (similar images) in which attributes similar to the attributes in the recognition image are captured are displayed. This allows the user to deepen their understanding of the attributes of the recognition image. Here, the similar images may be, for example, images similar in terms of the position on the map, or images having similarity other than attributes.

[0104] The model switching column 505 receives a designation for switching the classifier from the user. In the recognition result display column 503, the information output by the classifier designated by the model switching column 505 and its related information are displayed.

[0105] The communication menu 506 receives input from the user. When there is input from the user, the communication menu 506 transmits information on a predetermined task (information such as a shooting instruction, rescue instruction, etc.) to the terminal held by the shooter who performs shooting by the shooting system 101 or the worker at the shooting location. This task information includes, for example, information on a predetermined (e.g., highly confident) attribute among the attributes shown in the recognition result display column 503 (e.g., information indicating that the image has an attribute of flood or a collapsed house).

[0106] As described above, the image recognition support device 10 of the present embodiment recognizes an input image based on each of a plurality of types of classifiers and outputs recognition information, generates a pseudo label indicating the attributes of the input image based on each of the output recognition information, and generates a new label based on the generated pseudo label.

[0107] That is, since the image recognition support device 10 generates a new label based on the pseudo labels generated based on a plurality of types of classifiers, it is possible to gradually generate new labels with higher reliability from the pseudo labels obtained in the middle. Thus, according to the image recognition support device 10 of the present embodiment, it is possible to support the creation of a model for accurately recognizing the attributes in an image. For example, the need for manual label correction is reduced, and image recognition can be performed more simply and quickly.

[0108] Also, the image recognition support device 10 of the present embodiment inputs a specified image to each of a plurality of types of classifiers (each having a different tendency of recognition information of each attribute output according to the characteristic value of the attribute of the image) that output the reliability of each attribute that the input image has with respect to the classifier, calculates the reliability of each attribute that the specified image has based on the reliability of each attribute output from each classifier, and generates a pseudo label based on the calculated reliability. Then, the image recognition support device 10 generates a new label for each attribute of the input image based on the accuracy rate of each attribute of the pseudo label.

[0109] In this way, the image recognition support device 10 provides a plurality of classifiers with different configurations according to the characteristic values (appearance frequencies, etc.) of the attributes that may exist in the image, inputs a specified image to these classifiers and integrates them to generate a pseudo label, and generates a new label obtained by correcting the pseudo label based on its accuracy (accuracy rate).

[0110] As a result, labels (labels necessary for learning the learning model) for images capturing various attributes can be automatically and accurately generated. And using this, it is possible to generate an image recognition model capable of correctly recognizing the attributes reflected in the image.

[0111] Also, the image recognition support device 10 of the present embodiment generates recognition information based on each classifier and the weight coefficients associated with each attribute. Thereby, highly accurate image recognition can be performed according to the type of the image and the tendency of its subject.

[0112] Also, the image recognition support device 10 of the present embodiment sets, as a characteristic value regarding the attribute of the image, the frequency at which the attribute appears in the image. Thereby, image recognition can be performed according to the characteristics of the attribute of the image.

[0113] Note that the image recognition support device 10 of the present embodiment may set, as a characteristic value regarding the attribute of the image, information indicating the certainty of the recognition of the attribute by the classifier, that is, the reliability. Thereby, the accuracy of identifying the attribute by each classifier can be improved. Also, in this case, the step S1002 of calculating the frequency of each attribute can be omitted.

[0114] Also, the image recognition support device 10 of the present embodiment can facilitate data management related to image recognition by storing the designated image, the pseudo label, and the new label in association with each other in the integrated DB205.

[0115] Also, the image recognition support device 10 of the present embodiment can use the image as, for example, a SAR image or a CG image. Thereby, image recognition can also be supported for medical images, aerial images, synthetic images, etc. Also, even in cases where there is a tendency for biases in characteristic values (such as appearance frequency) regarding attributes to occur and where label information is likely to contain errors, like these images, the attributes in the image can be recognized with high accuracy without extremely performing additional manual label correction work.

[0116] In addition, the image recognition support device 10 of the present embodiment displays the recognition information of each attribute output by the classifier, and based on a designation from the user, transmits a work instruction to a terminal such as a worker associated with the attribute. Thereby, various operations according to the recognition status of the attributes of the image can be performed. For example, appropriate disaster relief and restoration can be carried out based on an image depicting a disaster situation.

[0117] In addition, the image recognition support device 10 of the present embodiment displays the recognition information for an attribute in which the pseudo label has a predetermined value. Thereby, for example, when the attribute is present in the image with a high probability, only that attribute can be provided to the user.

[0118] In addition, the image recognition support device 10 of the present embodiment displays the new label of each attribute and accepts a change of the new label from the user. Thereby, a more appropriate label can be set.

[0119] In addition, as a plurality of classifiers, the image recognition support device 10 of the present embodiment sets the learning weight of a low-frequency attribute, which is an attribute with a probability of presence in the image being equal to or less than a first threshold value, higher than the learning weights of other attributes, and an attribute with a probability of presence in the image being equal to or greater than a second threshold value which is a high-frequency attribute model in which the learning weight of the high-frequency attribute is set higher than the learning weights of other attributes. After including at least these, a designated image is input to each of the plurality of classifiers, and based on the reliability of each attribute output from each of the plurality of classifiers and the weight coefficient corresponding to the appearance frequency of each attribute associated with each classifier and each attribute, a total value of the reliability of each attribute possessed by the designated image is generated. In this way, by providing a plurality of classifiers in which the learning weight of each attribute is changed according to the appearance frequency of each attribute, and further calculating the integrated reliability using the weight coefficient corresponding to the appearance frequency of each attribute, a highly accurate pseudo label for the designated image can be generated.

[0120] Note that the present invention is not limited to the above-described embodiments, and can be implemented using any components without departing from the gist thereof. The embodiments and modifications described above are merely examples, and the present invention is not limited to these contents as long as the features of the invention are not impaired. Further, although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

[0121] For example, a part of each function provided in each device of the embodiment may be provided in another device, or a function provided in a separate device may be provided in the same device.

[0122] Also, in the present embodiment, as the characteristic values of the attributes, the appearance frequency and the reliability of each attribute appearing in the image are cited. However, other characteristic values, for example, the size of each attribute in the image, the content of the attribute (for example, adult or child), etc. may be adopted.

Description of Reference Numerals

[0123] 1 Image recognition system, 10 Image recognition support device, 202 Pseudo label generation unit, 203 New label generation unit

Claims

1. An image input unit that acquires an image, calculates the occurrence frequency of each attribute of the acquired image based on a predetermined image database respectively, inputs the acquired image to each of a plurality of classifiers that are learned using a loss function corresponding to the occurrence frequency so that the tendency of the recognition information of the attributes of the input image to be output is different from each other, based on the recognition information of each attribute output from each of the plurality of classifiers, generates integrated recognition information of each attribute of the acquired image, and a pseudo-label generation unit that generates a pseudo-label based on the integrated recognition information, An image recognition support device comprising: a new label generation unit that calculates the accuracy of each attribute of the pseudo-label and generates a new label for each attribute of the acquired image based on the calculated accuracy.

2. The pseudo-label generation unit generates the integrated recognition information based on the recognition information of each attribute output from each of the plurality of classifiers and a predetermined weight coefficient associated with each classifier and each attribute, The image recognition support device according to claim 1.

3. The image recognition support device according to claim 1, further comprising a storage unit that stores the acquired image, the pseudo-label of the image, and the new label of the image in association with each other.

4. The image recognition support device according to claim 1, wherein the image is a SAR image or a CG image.

5. The image recognition support device according to claim 1, further comprising a display unit that displays the recognition information of each output attribute and transmits information on a predetermined operation to a predetermined terminal associated with the attribute based on a designation from a user.

6. The image recognition support device according to claim 5, wherein the display unit displays recognition information about an attribute indicating that the integrated recognition information related to the pseudo label is a predetermined value or range.

7. The image recognition support device according to claim 5, wherein the display unit displays information about the new label corresponding to each attribute and accepts a change in the information about the new label from a user.

8. The image recognition support device according to claim 1, further comprising an image recognition model generation unit that generates a learned model that learns the relationship between the plurality of acquired images and the generated new labels respectively corresponding to the plurality of images, and outputs recognition information about each attribute of the input image.

9. As the plurality of classifiers, at least including a low-frequency attribute model that is a learning model in which the learning weight of a low-frequency attribute, which is an attribute with a probability of existence in an image being equal to or less than a first threshold value, is set higher than the learning weight of other attributes, and a high-frequency attribute model that is a learning model in which the learning weight of a high-frequency attribute, which is an attribute with a probability of existence in an image being greater than the first threshold value and equal to or greater than a second threshold value, is set higher than the learning weight of other attributes. The pseudo label generation unit inputs the acquired image to each of the plurality of classifiers, and generates a total value of the reliability of each attribute of the acquired image based on the reliability, which is the probability of existence of each attribute, output from each of the plurality of classifiers, and the weight coefficient corresponding to the appearance frequency of each attribute associated with each classifier and each attribute. The image recognition support device according to claim 1.

10. An information processing device An image input process for acquiring an image, Calculates the appearance frequency of each attribute of the image based on a predetermined image database respectively. For each classifier among a plurality of image recognition models that have been trained using a loss function corresponding to the frequency of occurrence of the attributes of the input image so that the tendencies of the recognition information of the attributes of the input image output according to the frequency of occurrence are different from each other, input the acquired image. Calculate the frequency of occurrence of each attribute of the image in the image based on a predetermined image database. For each classifier among a plurality of image recognition models that have different tendencies of the recognition information of the attributes of the input image output according to the frequency of occurrence of the attributes of the input image, input the acquired image. Based on the recognition information of each attribute of the acquired image output from each of the plurality of classifiers, generate integrated recognition information of each attribute of the acquired image, and perform a pseudo-label generation process of generating a pseudo-label based on the integrated recognition information. Calculate the accuracy for each attribute of the pseudo-label, and perform a new label generation process of generating a new label for each attribute of the acquired image based on the calculated accuracies. An image recognition support method for executing the above.

11. In an information processing apparatus, An image input process for acquiring an image, Calculate the frequency of occurrence of each attribute of the image in the image based on a predetermined image database. For each classifier among a plurality of image recognition models that have been trained using a loss function corresponding to the frequency of occurrence of the attributes of the input image so that the tendencies of the recognition information of the attributes of the input image output according to the frequency of occurrence are different from each other, input the acquired image. Calculate the frequency of occurrence of each attribute of the image in the image based on a predetermined image database. For each classifier among a plurality of image recognition models that have different tendencies of the recognition information of the attributes of the input image output according to the frequency of occurrence of the attributes of the input image, input the acquired image. Based on the recognition information of each attribute of the acquired image output from each of the plurality of classifiers, generate integrated recognition information of each attribute of the acquired image, and generate a pseudo label based on the integrated recognition information; a pseudo label generation process Calculate the accuracy for each attribute of the pseudo label, and based on the calculated accuracies, generate a new label for each attribute of the acquired image; a new label generation process An image recognition support program for causing the above to be executed.

Images