Image recognition assistance device, image recognition assistance method, and recording medium

Abstract

The present invention provides an image recognition assisting device, an image recognition assisting method, and a recording medium for assisting in the creation of models for recognizing attributes within an image with high precision. The image recognition assisting device (10) includes: an image input unit (201) for acquiring an image; a pseudo-label generation unit (202) for recognizing the acquired image based on each of a plurality of image recognition models and outputting recognition information, and generating pseudo-labels representing attributes of the acquired image based on the output recognition information; and a new label generation unit (203) for generating new labels based on the generated pseudo-labels.

Description

TECHNICAL FIELD

[0001] The present application relates to an image recognition assistance device, an image recognition assistance method, and a recording medium. BACKGROUND

[0002] By automatically recognizing a captured image, it is possible to determine the attributes of each subject in the image and to know the event recorded in the image. For example, aerial captured images and satellite images of a disaster site are useful as a means for remotely grasping the situation at the site, and in particular, by simultaneously recognizing a plurality of attributes in a wide-range image, it is possible to quickly grasp the disaster situation. With such an aim, in order to create a classifier that automatically recognizes and classifies each attribute captured in an image, it is necessary to prepare pairs of an image and a label (correct answer label) indicating all attributes within the image as learning data, and to cause the classifier to learn these pairs as patterns. Here, the pair of an image and a label is referred to as learning data.

[0003] However, in the preparation of learning data, it is difficult to correctly prepare correct answer labels for all attributes in a wide-range image. In particular, when a person sets a label for an image, there is a problem of mislabeling in which a label is assigned for an attribute that is not a correct answer, or a problem of missing labeling in which a label is not assigned despite the attribute. Furthermore, when learning patterns of images and labels, if a classifier is created using learning data including erroneous labels, there is a problem of a decrease in the accuracy of the classifier. Therefore, in order to correctly recognize attributes within an object image without a decrease in the accuracy of the classifier, it is necessary to correct labels assigned by a person.

[0004] As an image recognition assistance device and method for correcting a label set by a person, for example, Patent Literature 1 is known. In Patent Literature 1, a technology is described in which a reliability related to an attribute output from an image recognition section is acquired, and the acquired reliability is compared with label information set in advance in a display section to correct the label.

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2019-46095

[0006] In the technology of Patent Literature 1, it is necessary to compare label information set in advance with the reliability output from the image recognition section, and the user corrects the label manually. However, it takes a lot of time to correct a large number of images all by a person. On the other hand, in the case of automatically correcting a label based on the reliability output from the image recognition section, if the accuracy of the output reliability is not high enough, patterns can be learned based on erroneous label information, and a decrease in the accuracy of the image recognition section can still occur. SUMMARY

[0007] Therefore, the purpose of this invention is to provide an image recognition assisting device, an image recognition assisting method, and a recording medium that can assist in the creation of a model for recognizing attributes within an image with high precision.

[0008] One of the present inventions for solving the above-mentioned problems is an image recognition auxiliary device comprising: an image input unit for acquiring an image; a pseudo-label generation unit for recognizing the acquired image based on each of a plurality of image recognition models and outputting recognition information, and generating pseudo-labels representing attributes of the acquired image based on the output recognition information; and a new label generation unit for generating new labels based on the generated pseudo-labels.

[0009] Furthermore, one of the present inventions for solving the above-mentioned problems is an image recognition assistance method in which an information processing device performs: image input processing to acquire an image; pseudo-label generation processing to recognize the acquired image based on each of a plurality of image recognition models and output recognition information, and to generate pseudo-labels representing the attributes of the acquired image based on the output recognition information; and new label generation processing to generate new labels based on the generated pseudo-labels.

[0010] Furthermore, one aspect of the present invention for solving the aforementioned problems is a recording medium, which is a computer-readable recording medium recording an image recognition assistance program, wherein the image recognition assistance program enables a computer of an information processing device to perform: image input processing to acquire an image; pseudo-tag generation processing to recognize the acquired image based on each of a plurality of image recognition models and output recognition information, and to generate pseudo-tags representing the attributes of the acquired image based on the output recognition information; and new tag generation processing to generate new tags based on the generated pseudo-tags.

[0011] Invention Effects

[0012] According to the present invention, the new label generation unit (processing) generates new labels based on pseudo-labels generated by the pseudo-label generation unit (processing) based on multiple types of image recognition models, thus enabling the gradual generation of highly reliable new labels from pseudo-labels obtained along the way. Therefore, even without human verification (visual inspection, etc.), labels for highly reliable learning data can be generated. Furthermore, this allows for the generation of high-precision learning models based on data containing errors.

[0013] The issues, structures, and effects other than those mentioned above will become clearer through the following description of the implementation methods. Attached Figure Description

[0014] Figure 1This is a diagram illustrating the general structure of the image recognition system according to this embodiment.

[0015] Figure 2 This is an example of an illustration representing the original tag, pseudo tag, and new tag.

[0016] Figure 3 This is another example of the original label, pseudo label, and new label for an image.

[0017] Figure 4 This is a block diagram illustrating one example of the functions of an image recognition assistive device.

[0018] Figure 5 This is a diagram representing an example of information stored in a comprehensive database.

[0019] Figure 6 It is a detailed diagram illustrating the pseudo-tag generation process.

[0020] Figure 7 It is a detailed diagram illustrating the new label generation process.

[0021] Figure 8 This diagram illustrates an example of the hardware required for an image recognition assistive device.

[0022] Figure 9 This is a flowchart illustrating an example of image recognition auxiliary processing performed by an image recognition auxiliary device.

[0023] Figure 10 This is a flowchart illustrating the detailed process of pseudo-tag generation.

[0024] Figure 11 This is a diagram illustrating an example of a method for calculating reliability.

[0025] Figure 12 This is a flowchart illustrating the detailed process of generating new tags.

[0026] Figure 13 This diagram illustrates an example of the configuration of the operation screen displayed by an image recognition assistive device.

[0027] Explanation of reference numerals in the attached figures

[0028] 1 Image recognition system; 10 Image recognition auxiliary device; 202 Pseudo-label generation unit; 203 New label generation unit. Detailed Implementation

[0029] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. For clarity of explanation, the following descriptions and drawings will be appropriately omitted and simplified. Furthermore, the present invention is not limited to these embodiments, and all applications conforming to the spirit of the present invention are included within the technical scope of the present invention. Moreover, unless otherwise specified, the number of each constituent element mentioned can be either one or multiple.

[0030] <System Composition>

[0031] Figure 1 This diagram illustrates an outline of the configuration of the image recognition system 1 according to this embodiment. The image recognition system 1 comprises a camera system 101 for capturing images and an image recognition auxiliary device 10 for generating an image recognition model based on the images captured by the camera system 101. The camera system 101 and the image recognition auxiliary device 10 are communicatively connected, for example, via a wired or wireless network 5 such as a LAN (Local Area Network), WAN (Wide Area Network), the Internet, or a leased line.

[0032] The photography system 101 comprises one or more photography devices (cameras) for capturing images. The photography device can be, for example, a human-held photography device, a device fixed to the ground, a photography device installed on a vehicle moving on the ground, or a photography device installed on a drone or airplane.

[0033] The images captured by the camera system 101 include images of one or more subjects (objects), and the user can associate each object with a specific attribute from a pre-listed set of attributes (types). Attributes can be, for example, man-made objects such as people, buildings, vehicles, or roads, natural objects such as the sea or rivers, or attributes representing the state of objects or people, such as floods, building collapses, traffic congestion, or dense crowds.

[0034] Furthermore, the images can be either color or monochrome. In addition to images obtained from a camera, the images can also be SAR (Synthetic Aperture Radar), CG (Computer Graphics), or other types of images acquired beforehand. Furthermore, metadata can be included with each image.

[0035] Image recognition assist device 10 acquires images captured by camera system 101. Image recognition assist device 10 generates multiple classifiers (learned models) with different configurations for recognizing the attributes of the image. Furthermore, image recognition assist device 10 inputs a user-specified image (specified image) into each of the generated classifiers, and generates new labels (new labels) by comparing the values ​​of each label (pseudo-label) obtained based on the output values ​​of each classifier with the values ​​of labels (original labels) pre-set by the user for the specified image, according to each attribute.

[0036] The newly generated labels in this way correctly reflect the information about the attributes of the specified image.

[0037] Then, the image recognition assist device 10 generates an image recognition model by learning the relationship between multiple specified images and new labels through machine learning.

[0038] <Original tags, pseudo tags, and new tags>

[0039] Figure 2 This is an example of an illustration showing the original label, pseudo label, and new label. As shown in the illustration, label information 402 (original label, pseudo label, and new label) is set for image 401.

[0040] Label information 402 is information indicating the probability that image 401 possesses a certain attribute at a learning time point (period) in the machine learning processing of the classifier (described later), or information indicating the presence or absence of a certain attribute in image 401. The original label 402a in label information 402 is pre-set by the user or others and may contain errors. The pseudo-label 402b is automatically set by the pseudo-label generation unit 202 (described later). The new label 402c is automatically set by the new label generation unit 203 (described later) based on the original label 402a and the pseudo-label 402b.

[0041] Furthermore, regarding the description of the image label information 402, for example, "x123" indicates the original label with attribute "3" set during period "2" (number of machine learning trials) related to the image with ID "1". Additionally, "y342" indicates a pseudo-label with attribute "2" set during period "4" related to the image with ID "3". Furthermore, "z567" indicates a new label with attribute "7" set during period "6" related to the image with ID "5". While pseudo-labels and new labels may differ for each period, the original label is common across all periods.

[0042] then, Figure 3This diagram illustrates another example of original labels, pseudo labels, and new labels for an image. As shown, label information 404 (original label, pseudo label, and new label) is set for image 403. The original label 404a, pseudo label 404b, and new label 404c in label information 404 are the same as the original label 402a, pseudo label 402b, and new label 402c mentioned above. Furthermore, in label information 404, the original label 404a, pseudo label 404b, and new label 404c are the same as those in image 402a, pseudo label 402b, and new label 402c mentioned above. Figure 2 Unlike other methods, coordinate information 405, representing the position of the subject in image 403, is added.

[0043] As shown in the label information 402 and 404 above, the image recognition assist device 10 can correctly identify all the attributes contained in the image specified by the user for image recognition by creating new labels 402c and 404c that are more correct than the original labels 402a and 404a that may contain errors, and using them in the image recognition model.

[0044] Image recognition auxiliary device

[0045] then, Figure 4 This is a block diagram illustrating one example of the functions of the image recognition assistive device 10. The image recognition assistive device 10 includes functional units (programs) such as an image input unit 201, a pseudo-label generation unit 202, a new label generation unit 203, a classifier storage unit 204, and an image recognition model generation unit 206. Furthermore, the image recognition assistive device 10 stores a comprehensive DB 205 (DB: database).

[0046] The image input unit 201 acquires images captured by the camera system 101 and stores the acquired images in the integrated database 205. Furthermore, the image input unit 201 inputs each image to the pseudo-tag generation unit 202.

[0047] The pseudo-label generation unit 202 recognizes the image obtained by the image input unit 201 based on each of the multiple types of image recognition models (classifiers), outputs recognition information, and generates pseudo-labels representing the attributes of the obtained image based on the output recognition information.

[0048] Specifically, the pseudo-label generation unit 202 first generates and stores multiple types of classifiers. These multiple types of classifiers are input to the image and output recognition information for each attribute possessed by the image. In this embodiment, the recognition information is assumed to be the reliability of the probability (likelihood) that the image possesses that attribute.

[0049] Each classifier is generated based on the images stored in the integrated DB205 and the labels of those images (original labels, and new labels, which will be described later if they exist), and is generated such that the recognition information of each attribute output according to the characteristic values ​​related to each attribute of the image tends to be different.

[0050] Furthermore, in this embodiment, it is assumed that the characteristic value of each attribute is the frequency of occurrence of each attribute (the probability of the attribute existing in the image). The frequency of occurrence here can be the frequency of occurrence of each attribute in all images currently captured by the imaging system 101, the frequency of occurrence of each attribute in a specific group of images, or other statistically derived frequencies of occurrence.

[0051] Furthermore, each classifier is based on a learned model generated by deep learning. Examples of such classifiers include convolutional neural networks, which consist of multi-layered information networks.

[0052] Furthermore, the pseudo-label generation unit 202 identifies the attributes of the image by inputting a specified image from the image input unit 201 into multiple classifiers, and outputs the results as recognition information.

[0053] Furthermore, the pseudo-label generation unit 202 calculates the comprehensive recognition information (hereinafter also referred to as comprehensive reliability) of each attribute of the specified image based on the recognition information (reliability) of each attribute output from multiple classifiers and the corresponding prescribed coefficients (weight coefficients) established with each combination of classifiers and attributes, and generates pseudo-labels obtained by performing a prescribed transformation on the specified image.

[0054] The pseudo-label generation unit 202 inputs the comprehensive identification information of each attribute and the pseudo-label into the new label generation unit 203.

[0055] The new tag generation unit 203 generates new tags based on the pseudo tags generated by the pseudo tag generation unit 202.

[0056] Specifically, the new label generation unit 203 calculates the correctness (label accuracy) of each attribute of the pseudo-label based on the comprehensive recognition information of each attribute input from the pseudo-label generation unit 202. Based on the calculated correctness, the new label generation unit 203 generates new labels for each attribute of the specified image by correcting the pseudo-labels for each attribute of the pseudo-labels.

[0057] In addition, the generated new labels are stored in the comprehensive DB205. Furthermore, the generated new labels are repeatedly used in the machine learning of each classifier performed by the pseudo-label generation unit 202.

[0058] The classifier storage unit 204 stores each classifier. Furthermore, the classifier storage unit 204 stores information on the correctness of pseudo-labels, the learning parameters of each classifier, and the recognition information of each attribute obtained by each classifier. This information is used, for example, during the generation of pseudo-labels by the classifiers or during the classifiers' machine learning process.

[0059] DB205 comprehensively stores the original labels, label information, shooting time, and map information for each image. For example, DB205 stores the ID, period, original label, pseudo label, and new label for each image.

[0060] The image recognition model generation unit 206 generates a learned model (image recognition model) for image attribute recognition based on the images stored in the comprehensive database 205 and the new labels for these images. For example, the image recognition model generation unit 206 generates a learned model that takes an image as input and outputs recognition information (reliability, etc.) of each attribute of the image by learning the relationship between multiple specified images and the new labels corresponding to each specified image. In addition, this learned model is configured, for example, as a convolutional neural network with multiple layers.

[0061] (Comprehensive DB)

[0062] here, Figure 5 This diagram illustrates an example of the information stored in the integrated DB205. The integrated DB205 includes data items such as ID 302 (setting the identifier for each image), time of capture 303 (setting the capture date and time for each image), period 304 (setting the period, specifically, the number of trials in which the processing S1002 to S1007 described later was performed), original label 305 (setting the original label for each attribute of each image), pseudo label 306 (setting the pseudo label for each attribute of each image), and new label 307.

[0063] Additionally, for the original label 305, pseudo label 306, and new label 307, identification information (reliability, etc.) for multiple attributes contained in the image is set. Furthermore, it is assumed that the user has pre-set the original label 305. Moreover, the data items described here are just one example; for instance, they could also include information such as image metadata.

[0064] (Pseudo-tag generation department)

[0065] then, Figure 6 This diagram illustrates the details of the pseudo-tag generation unit 202. The pseudo-tag generation unit 202 includes functional units (programs) such as a collection object selection unit 601, an attribute weight estimation unit 602, an attribute score collection processing unit 603, and a pseudo-tag generation unit 604.

[0066] The set object selection unit 601 selects multiple classifiers to be used for generating pseudo-labels.

[0067] The attribute weight estimation unit 602 sets values ​​(weight coefficients) related to the learning weights of each attribute in the machine learning of each classifier based on the characteristic values ​​(frequency of occurrence) of each attribute in the image synthesized from DB205. Furthermore, in this embodiment, it is assumed that the weight coefficients are automatically calculated based on the characteristic values ​​of each attribute and the hyperparameters of each attribute in the classifier.

[0068] The attribute score set processing unit 603 calculates the recognition information (overall reliability) for each attribute of the image based on the output value (reliability for each attribute) of the image input to each classifier selected by the set object selection unit 601 and the weight coefficient set by the attribute weight estimation unit 602.

[0069] The pseudo-label generation unit 604 transforms the overall reliability of each attribute calculated by the attribute score set processing unit 603 into pseudo-labels. For example, the pseudo-label generation unit 604 transforms the overall reliability, which is a continuous value, into the value of the pseudo-label (e.g., 0 or 1), which is a discrete value.

[0070] (New Tag Generation Department)

[0071] then, Figure 7 This diagram illustrates the details of the new label generation unit 203. The new label generation unit 203 includes functional units (programs) such as a pseudo-label processing unit 801, an attribute threshold setting unit 802, a label fusion unit 803, and a new label transformation unit 804.

[0072] The fake tag processing unit 801 performs the same processing as the fake tag generation unit 604 (conversion from reliability to tag value). The fake tag processing unit 801 performs this processing if the above processing is not performed in the fake tag generation process.

[0073] The attribute threshold setting unit 802 sets a parameter (threshold) for each attribute of a specified image by comparing the pseudo-labels generated by the pseudo-label generation unit 202 with the original labels. This parameter determines whether to use the pseudo-labels as new labels for the specified image. Specifically, the attribute threshold setting unit 802 sets a high value as the threshold for a certain attribute if the recognition accuracy (the probability that the value of the original label and the value of the pseudo-label for a certain attribute are the same) is high, and sets a low value as the threshold for a certain attribute if the recognition accuracy is low.

[0074] The new tag fusion unit 803 generates new tags for each attribute based on the pseudo tags for each attribute generated by the pseudo tag processing unit 801 and the threshold values ​​for each attribute generated by the attribute threshold setting unit 802.

[0075] The new label transformation unit 804 transforms the labels using the same method as the pseudo label generation unit 604. For example, if the new label values ​​for each attribute generated by the new label fusion unit 803 are reliability values, and the reliability of a certain attribute is 0.5 or higher, then a "1" representing that attribute in the image will be set as the new label value; if the reliability of a certain attribute is less than 0.5, then a "0" representing that the image does not have that attribute will be set as the new label value. Furthermore, the value transformation method described here is just one example; any other method can be used.

[0076] here, Figure 8 This diagram illustrates an example of the hardware included in the image recognition assistive device 10. The image recognition assistive device 10 includes: a processing unit 103 such as a CPU (Central Processing Unit), DSP (Digital Signal Processor), GPU (Graphics Processing Unit), or FPGA (Field-Programmable Gate Array); a storage device 104 composed of a storage device or storage medium such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), or SSD (Solid State Drive); a display device 105 composed of a liquid crystal display or an organic EL (Electro-Luminescence) display; an input device 106 composed of a mouse or keyboard; and a communication device 102 composed of a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, or a serial communication module.

[0077] In addition, the display device 105 displays the images captured by the photography system 101 and information about each label (fake label, new label, etc.).

[0078] In addition, the input device 106 accepts input from the user. For example, the input device 106 accepts input from the user regarding the switching of the classifier displayed on the display device 105, and accepts input from the user regarding the setting or correction of labels for images captured by the camera system 101.

[0079] The functions of the image recognition assistive device 10 are implemented by the processing device 103 reading and executing the program stored in the storage device 104. Furthermore, the aforementioned program can be recorded on a recording medium and distributed, for example.

[0080] Next, the processing performed by the image recognition assist device 10 will be explained.

[0081] <Image Recognition-Assisted Processing>

[0082] Figure 9 This is a flowchart illustrating an example of image recognition assistance processing performed by the image recognition assistance device 10. This processing begins, for example, when an image captured by the camera system 101 is stored in the integrated DB 205, or when a user provides a specified input to the image recognition assistance device 10.

[0083] First, the pseudo-label generation unit 202 calculates the frequency of occurrence of each attribute based on the original labels of each image stored in the comprehensive DB 205, and stores the results (step S1001).

[0084] In this embodiment, the pseudo-tag generation unit 202 determines the attribute of an image with a frequency lower than a preset first threshold as a "low-frequency attribute" and sets the image as a "low-frequency image", determines the attribute of an image with a frequency of ...

[0085] Furthermore, the pseudo-label generation unit 202 generates a batch list containing information about all images stored in the comprehensive DB 205 (step S1002). This batch list includes, for example, index information for each image and information about the image's label (original label or new label). Additionally, this batch list information is information commonly used in deep learning.

[0086] Furthermore, the pseudo-label generation unit 202 selects an image from the batch list generated in step S1002 and extracts information such as the original label and new label of the selected image (step S1003). Additionally, when step S1003 is initially executed, there is no information about the new label.

[0087] The pseudo-tag generation unit 202 performs a pseudo-tag generation process (step S1004) to create pseudo-tags based on the image selected in step S1003 and the information of each tag. Details of the pseudo-tag generation process S1004 will be described later.

[0088] In this embodiment, the pseudo-label generation unit 202 generates pseudo-labels by generating multiple different classifiers corresponding to the frequency of occurrence of attributes. Specifically, the pseudo-label generation unit 202 generates a low-frequency emphasis model that prioritizes low-frequency images among high-frequency, medium-frequency, and low-frequency images, a medium-frequency emphasis model that prioritizes medium-frequency images, and a high-frequency emphasis model that prioritizes high-frequency images. In particular, the pseudo-label generation unit 202 generates classifiers by maximizing the learning weights for each of the high-frequency, medium-frequency, and low-frequency images.

[0089] In this way, by setting up classifiers with multiple different characteristics, higher image recognition accuracy can be achieved compared to using only a single classifier.

[0090] The new label generation unit 203 performs a new label generation process S1005 (step S1005) to generate new labels based on the pseudo-labels for each attribute generated by the pseudo-label generation unit 202 and the correctness (positive resolution rate) of the pseudo-labels for each attribute of each classifier. Furthermore, the generated new labels replace the original labels in the next period and are used for classifier learning. Details of the new label generation process S1005 will be described later.

[0091] The new label generation unit 203 checks whether all images have been selected from the batch list (step S1006). If there are images not selected from the batch list (step S1006: No), the pseudo-label generation unit 202 continues the machine learning of the neural network by selecting one of these images and repeating the processing after step S1003. If there are no images not selected from the batch list (step S1006: Yes), the processing in step S1007 is performed.

[0092] In step S1007, the new tag generation unit 203 checks whether a preset period (number of learning iterations) has been reached. If the period has not been reached (step S1007: No), the pseudo tag generation unit 202 repeats the processing after step S1002. If the period has been reached (step S1007: Yes), the processing in step S1008 is performed.

[0093] In step S1008, the new label generation unit 203 stores the pseudo labels and new labels of each image and the recognition results of each attribute obtained by each classifier into the comprehensive DB205.

[0094] Furthermore, the new label generation unit 203 determines whether the pre-specified number of repetitions (iterations) has been reached (step S1009). If no iteration has been reached (step S1009: No), the pseudo-label generation unit 202 repeats the processing after step S1002. If an iteration has been reached (step S1009: Yes), the pseudo-label generation unit 202 performs the processing of step S1010.

[0095] Then, the image recognition assist device 10 performs machine learning on the combination of each image and the new label stored in the integrated DB205 to learn the relationship between them, and generates an image recognition model (step S1010). By inputting the image for which the desired attribute is to be recognized into the image recognition model, the user can obtain the attribute of the image.

[0096] The image recognition-assisted processing is now complete.

[0097] <Pseudo-tag generation and processing>

[0098] Figure 10 This is a flowchart illustrating the detailed process of pseudo-tag generation and processing S1004.

[0099] The pseudo-label generation unit 202 obtains the reliability of each attribute of the specified image output from each classifier by inputting a specified image into each classifier (step S701). In addition, the reliability is, for example, expressed in the range of 0 to 1, representing the probability of each attribute being present in the specified image.

[0100] The set object selection unit 601 selects a classifier for generating pseudo-labels from all the classifiers stored in the classifier storage unit 204 (step S702).

[0101] In this case, the set object selection unit 601 can select all classifiers, or select only the classifiers with good recognition results (for example, the classifier that correctly identified the attribute with a specified probability or higher in the current processing), or select classifiers according to other specified criteria.

[0102] The attribute weight estimation unit 602 sets the weight coefficients (during learning) of each attribute in each classifier based on the frequency of occurrence of each attribute obtained in step S1001 (step S703).

[0103] For example, the attribute weight estimation unit 602 sets the weight coefficient of the output for low-frequency images in the high-frequency importance model to 0.3 (a low value), and sets the weight coefficient of the output for low-frequency images in the low-frequency importance model to 0.7 (a high value). The weight coefficient is automatically determined, for example, by setting it as a hyperparameter (the frequency of occurrence of the attribute) in the neural network.

[0104] The attribute score set processing unit 603 calculates the overall reliability based on the classifier selected in step S702 and the weight coefficients of each attribute set in step S703 (step S704).

[0105] Figure 11 This is a diagram illustrating an example of a reliability calculation method. As shown in the diagram, assuming there are three classifiers: a low-frequency importance model 51, a medium-frequency importance model 52, and a high-frequency importance model 53, the attributes that are most valued and learned in the low-frequency importance model 51 (i.e., low-frequency attributes) are attributes 1 and 2; the attributes that are most valued and learned in the medium-frequency importance model 52 (i.e., medium-frequency attributes) are attributes 3 and 4; and the attributes that are most valued and learned in the high-frequency importance model 53 (i.e., high-frequency attributes) are attributes 5 and 6.

[0106] First, in step S701, the attribute score set processing unit 603 inputs a specified image into each classifier (low-frequency importance model 51, mid-frequency importance model 52, and high-frequency importance model 53) and calculates the recognition results of each classifier for the specified image (reliability 54 for low-frequency attributes 1 and 2, reliability 55 for mid-frequency attributes 3 and 4, and reliability 56 for high-frequency attributes 5 and 6). Furthermore, for each attribute, the attribute score set processing unit 603 multiplies the reliability 54, 55, and 56 calculated by each classifier with the weight coefficient 57 set according to the classifier and attribute, and sums their multiplied values ​​to obtain a comprehensive reliability 58.

[0107] Furthermore, since the low-frequency attribute emphasis model can identify low-frequency attributes with higher accuracy compared to other models, the weight coefficient (0.7) for low-frequency images in the low-frequency attribute emphasis model is set to a larger value than the weight coefficients for low-frequency images in other models (high-frequency attribute emphasis model: 0.1 and mid-frequency attribute emphasis model: 0.2). The same applies to other attributes; in the mid-frequency attribute emphasis model, a high weight coefficient is set for mid-frequency attributes.

[0108] Here, examples illustrating how to set the weight coefficients for each classifier are provided. The loss function used in machine learning for each classifier can be...

[0109] Focal Loss(FL(p t ))=-(1-p t )γ×log(p t )

[0110] The structure is such that the larger the coefficient γ in Focal Loss, the more difficult it is to identify, meaning it emphasizes data with low-frequency attributes. For example, if the coefficient γ1 of the low-frequency attribute emphasis model is pre-set to 3.0, the coefficient γ2 of the mid-frequency attribute emphasis model is pre-set to 2.0, and the coefficient γ3 of the high-frequency attribute emphasis model is pre-set to 1.0, the coefficients can be set using the values ​​of a normal distribution with a mean of 0 and a variance δ. Here, the value of the normal distribution refers to the value of the probability density function corresponding to the input variable x. For example, the weight coefficients of the low-frequency attribute emphasis model for low-frequency images (in...) Figure 11 In the example, 0.7) represents the probability density function value when x = 0 in a normal distribution. The mid-frequency attributes emphasize the model's weighting coefficients for images with low-frequency attributes (in...). Figure 11 In the example, 0.2) represents the probability density function value in the case of a normal distribution where x = |γ1 - γ2|. High-frequency attributes emphasize the model's weighting coefficients for low-frequency images (in...). Figure 11 In the example, 0.1) represents the value of the probability density function for the case of a normal distribution x = |γ1 - γ3|. By standardizing these values ​​(e.g., making the sum of these values ​​equal to 1), the weight coefficients can be calculated (in...). Figure 11 In the example, it is 0.7 + 0.2 + 0.1 = 1. Therefore, it is possible to automatically set weight coefficients for attributes that tend to have higher calculated reliability for each classifier.

[0111] Next, as Figure 10 As shown, the pseudo-label generation unit 604 generates pseudo-labels based on the reliability of each attribute calculated in step S704 (step S705).

[0112] Specifically, the pseudo-label generation unit 604 transforms each reliability level into a discrete value. For example, if the reliability value of a certain attribute is 0.5 or higher, the pseudo-label generation unit 604 sets the pseudo-label value of that attribute to "1" indicating the presence of the attribute; if the reliability value of a certain attribute is less than 0.5, the pseudo-label value of that attribute is set to "0" indicating the absence of the attribute. Furthermore, the pseudo-label values ​​described here are just one example; values ​​can be set using any other method.

[0113] The pseudo-tag generation unit 604 stores the pseudo-tag information generated in step S705 into the integrated DB 205 (step S706). Specifically, the pseudo-tag generation unit 604 sets the values ​​of the pseudo-tags for each attribute of the pseudo-tags 306 in the integrated DB 205.

[0114] As described above, the pseudo-label generation unit 202 generates and saves pseudo-labels by integrating and transforming the recognition results of each attribute of each classifier (learned model).

[0115] <New Tag Generation Processing>

[0116] Figure 12 This is a flowchart illustrating the details of the new label generation process S1005.

[0117] The new label generation unit 203 obtains the pseudo label generated by the pseudo label generation process S1004 and inputs the obtained pseudo label into the pseudo label processing unit 801 (step S901).

[0118] Furthermore, the new label generation unit 203 calculates the correctness rate as the recognition result for each attribute (step S902). For example, the new label generation unit 203 calculates the correctness rate for each attribute of the image by comparing the overall reliability based on each classifier calculated in the pseudo-label generation process S1004 with the value of the original label for each attribute. However, the method for calculating the correctness rate described here is just one example; the new label generation unit 203 can also evaluate the correctness of the pseudo-labels for each attribute of the image using any other arbitrary method.

[0119] In addition, if the pseudo-tag processing unit 801 does not perform the transformation in step S705 on the pseudo-tag value obtained in step S901, it sets the pseudo-tag value for each attribute in the same way as in step S705 (step S903).

[0120] The attribute threshold setting unit 802 sets a threshold for each attribute based on the positive solution rate calculated in step S902 (step S904).

[0121] For example, if the recognition accuracy of attribute 1 in the image is 10%, the attribute threshold setting unit 802 can consider the accuracy of the pseudo-label of attribute 1 to be low, so it sets a threshold such that the proportion of pseudo-labels used is 0.1 times that of the original labels. If the accuracy of attribute 1 in the image is 95%, the accuracy of the pseudo-labels can be considered high, so it sets a threshold such that the proportion of pseudo-labels used is 1 times that of the original labels. The attribute threshold setting unit 802 performs these settings for all attributes. In addition, the attribute threshold setting unit 802 can set the threshold based on input from the user, or it can automatically determine the threshold based on the value of each recognition accuracy.

[0122] The new tag fusion unit 803 generates new tags for each attribute based on the pseudo tags of each attribute calculated in step S902 (step S903) and the threshold values ​​of each attribute set in step S904 (step S905).

[0123] For example, for attributes 1 to 5, with the original label being (1, 1, 0, 0, 1), the pseudo label being (1, 0, 0, 1, 1), and the threshold being (1, 1, 1, 0, 1), and the accuracies of attributes 1 to 5 being 80%, 70%, 90%, 20%, and 95% respectively, since the recognition accuracy of the pseudo label for attribute 4 is low, the new label fusion unit 803 sets the new label for attribute 4 as the original label (not the pseudo label). Thus, the new label fusion unit 803 calculates the new labels for attributes 1 to 5 using the threshold values ​​of each attribute as (1+1) / 2 = 1, (1+0) / 2 = 0.5, (0+0) / 2 = 0, 0, and (1+1) / 2 = 1 respectively.

[0124] The new label transformation unit 804, like the pseudo label generation unit 604, transforms the new label calculated in step S905 (step S906).

[0125] For example, when the new label transformation unit 804 sets the new label to a discrete value, if the value of the new label calculated in step S905 is 0.5 or higher, it sets "1" to indicate that the image has the corresponding attribute; if the value of the new label is less than 0.5, it sets "0" to indicate that the image does not have the corresponding attribute. The method for transforming the new label is not limited to the method described here, and various other methods can be used.

[0126] The classifier storage unit 204 stores the new label generated in step S906 into the comprehensive DB 205. The classifier storage unit 204 establishes a correspondence between the image, each attribute, the new label, the classifier, and the period and stores it into the comprehensive DB 205.

[0127] As described above, the image recognition assist device 10 generates and saves new labels for each attribute of the image by integrating the pseudo-labels of each attribute of each classifier.

[0128] <Operation Screen>

[0129] Figure 13 This diagram illustrates an example of the configuration of the operation screen 150 displayed by the image recognition assistive device 10. The operation screen 150 includes a recognition object image display bar 501, a recognition object map display bar 502, a recognition result display bar 503, a similar image display bar 504, a model switching bar 505, and a contact menu 506.

[0130] In the object recognition image display area 501, an image (recognition image) showing the attributes identified by the classifier is displayed. Alternatively, the location of the identified attributes (objects, etc.) can be displayed on the image in the object recognition image display area 501 or at other designated locations.

[0131] In the map display panel 502, information such as the latitude and longitude of the location from which the identified image was obtained, along with a map of the region, is displayed. This map is not limited to two dimensions; it can also be displayed in three dimensions if elevation information is available.

[0132] The recognition result display panel 503 shows information about each attribute output from the classifier and its associated information (reliability, pseudo-label, correct answer rate, new label, etc. for each attribute). Alternatively, the recognition result display panel 503 may not display all attribute information, but only information about attributes that conform to certain criteria or are specified by the user. For example, it may only display the pseudo-label, correct answer rate, and new label information for attributes with a reliability value above a certain threshold.

[0133] Additionally, on the operation screen 150, a new label correction field 507 can be set for the input of the new label correction displayed by the user.

[0134] In the similar image display section 504, other images (similar images) that have properties similar to those in the identified image are displayed. This allows the user to deepen their understanding of the properties of the identified image. Here, similar images can be, for example, images that are similar in terms of location on a map, or images that have similarities in terms of properties other than those of the identified image.

[0135] The model switching bar 505 specifies the classifier switch from the user's input. The recognition result display bar 503 displays the output information of the classifier specified by the model switching bar 505 and its associated information.

[0136] The communication menu 506 accepts input from the user. If there is input from the user, the communication menu 506 sends the prescribed task information (photography instructions, rescue instructions, etc.) to the terminal maintained by the photographer taking pictures through the photography system 101 or the operator at the shooting location. The task information includes, for example, information on the prescribed attributes (e.g., high reliability) displayed in the recognition result display bar 503 (e.g., information indicating that the image has flood attributes or collapsed building attributes).

[0137] As described above, the image recognition assist device 10 of this embodiment recognizes the input image based on each of multiple types of classifiers and outputs recognition information. Based on the output recognition information, it generates pseudo-labels representing the attributes of the input image and generates new labels based on the generated pseudo-labels.

[0138] That is, the image recognition assist device 10 generates new labels based on pseudo-labels generated by classifiers of multiple types, thus enabling the gradual generation of new, highly reliable labels based on pseudo-labels obtained along the way. In this way, the image recognition assist device 10 according to this embodiment can assist in the creation of models that accurately recognize attributes within an image. For example, the need for manual label correction is reduced, making image recognition simpler and faster.

[0139] Furthermore, regarding the classifier, the image recognition assisting device 10 of this embodiment inputs a specified image to multiple classifiers of various types (each classifier outputting recognition information for each attribute based on the characteristic values ​​of the image's attributes) that output the reliability of each attribute possessed by the input image. Based on the reliability of each attribute output from each classifier, the device calculates the reliability of each attribute possessed by the specified image and generates pseudo-labels based on the calculated reliability. Then, based on the positive resolution rate of each attribute of the pseudo-labels, the image recognition assisting device 10 generates new labels for each attribute of the input image.

[0140] In this way, the image recognition assist device 10 sets up multiple classifiers based on the characteristic values ​​(frequency of occurrence, etc.) of the attributes that may exist in the image. By inputting a specified image into these classifiers and combining them, pseudo-labels are generated, and new labels are generated after correcting the pseudo-labels based on their accuracy (correct resolution rate).

[0141] The result is the ability to automatically and accurately generate labels for images with a wide variety of attributes (labels needed for the learning model to learn). Furthermore, these labels can be used to generate image recognition models that can correctly identify the attributes captured within an image.

[0142] Furthermore, the image recognition assist device 10 of this embodiment generates recognition information based on weight coefficients established with respect to each classifier and each attribute. This enables high-precision image recognition that corresponds to the type of image and the orientation of the subject.

[0143] Furthermore, in the image recognition assist device 10 of this embodiment, the frequency of occurrence of an attribute in the image is set as a characteristic value relating to the image's attributes. Therefore, image recognition corresponding to the characteristics of the image's attributes can be performed.

[0144] Furthermore, in the image recognition assist device 10 of this embodiment, the characteristic values ​​representing the attributes of the image can also be set as information representing the probability of the classifier recognizing the attribute, i.e., reliability. This improves the accuracy of attribute identification by each classifier. In addition, in this case, the step S1002 of calculating the frequency of each attribute can be omitted.

[0145] Furthermore, the image recognition assist device 10 of this embodiment makes image recognition data management easier by establishing and storing the specified image, pseudo-label, and new label in the integrated DB205.

[0146] Furthermore, the image recognition assistance device 10 of this embodiment can recognize images such as SAR images or CG images. Therefore, it can also assist in image recognition for satellite images, aerial images, or composite images. Moreover, even in situations where biases in attribute characteristic values ​​(such as frequency of occurrence) are prone to occur, and where errors are easily included in the tag information, attributes in the image can be identified with high accuracy with minimal manual tag correction.

[0147] Furthermore, the image recognition assist device 10 of this embodiment displays recognition information for each attribute output by the classifier, and sends work instructions to terminals of operators or similar entities that have established correspondence with the attribute, based on user specifications. This enables the execution of various services corresponding to the recognition status of image attributes. For example, it allows for appropriate disaster relief and recovery based on images of disaster situations.

[0148] Furthermore, the image recognition assist device 10 of this embodiment displays recognition information about attributes of pseudo-tags that have predetermined values. Therefore, for example, if the attribute exists in the image with a high probability, only that attribute can be provided to the user.

[0149] Furthermore, the image recognition assistive device 10 of this embodiment displays new labels for each attribute and accepts changes to the new labels from the user. This allows for the setting of more appropriate labels.

[0150] Furthermore, the image recognition assistive device 10 of this embodiment includes at least a low-frequency attribute model that sets the learning weight of attributes (low-frequency attributes) with a probability of less than a first threshold in the image to be higher than the learning weight of other attributes, and a high-frequency attribute model that sets the learning weight of attributes (high-frequency attributes) with a probability of more than a second threshold in the image to be higher than the learning weight of other attributes. A specified image is input to each of the multiple classifiers, and based on the reliability of each attribute output from each of the multiple classifiers and the weight coefficients corresponding to the frequency of occurrence of each attribute corresponding to each classifier and each attribute, a total reliability value for each attribute of the specified image is generated. Thus, by setting multiple classifiers that change the learning weight of each attribute according to its frequency of occurrence, and then calculating the overall reliability using weight coefficients corresponding to the frequency of occurrence of each attribute, highly accurate pseudo-labels for the specified image can be generated.

[0151] Furthermore, the present invention is not limited to the embodiments described above, and can be implemented using any constituent elements without departing from its spirit. The embodiments and modifications described above are merely examples, and the present invention is not limited to these contents as long as the features of the invention are not impaired. In addition, various embodiments and modifications have been described above, but the present invention is not limited to these contents. Other forms that can be conceived within the scope of the technical concept of the present invention are also included within the scope of the present invention.

[0152] For example, some of the functions of each device in the implementation method can also be provided in other devices, or the functions of other devices can be provided in the same device.

[0153] Furthermore, in this embodiment, the frequency and reliability of each attribute appearing in the image are given as characteristic values ​​of the attribute, but other characteristic values ​​may also be used, such as the size of each attribute in the image, the content of the attribute (e.g., adult or child).

Claims

1. An image recognition auxiliary device, wherein, have: Image input unit acquires an image; The pseudo-label generation unit recognizes the acquired image based on each of the multiple types of image recognition models and outputs recognition information. Based on the output recognition information, it generates pseudo-labels representing the attributes of the acquired image. The new tag generation unit generates new tags based on the pseudo tags generated above. The pseudo-label generation unit inputs the acquired image into multiple classifiers, wherein the multiple classifiers output recognition information for each attribute of the input image based on characteristic values ​​related to the attributes of the input image, and the pseudo-label generation unit generates comprehensive recognition information for each attribute of the acquired image based on the recognition information for each attribute output from the multiple classifiers, and generates pseudo-labels based on the comprehensive recognition information. The aforementioned new label generation unit calculates the correctness of each attribute of the aforementioned pseudo-labels, and based on the calculated correctness, generates new labels for each attribute of the acquired image. As the aforementioned classifiers, at least a low-frequency attribute model and a high-frequency attribute model are included. The low-frequency attribute model is a learning model that sets the learning weight of attributes with a probability of being present in the image below the first threshold (i.e., low-frequency attributes) to be higher than the learning weight of other attributes. The high-frequency attribute model is a learning model that sets the learning weight of attributes with a probability of being present in the image above the second threshold (which is greater than the first threshold) to be higher than the learning weight of other attributes. The pseudo-label generation unit generates a total value of the reliability of each attribute of the obtained image based on the reliability of the probability of each attribute being output from the multiple classifiers respectively, and the weight coefficients corresponding to the occurrence frequency of each attribute established with each classifier and each attribute.

2. The image recognition auxiliary device as described in claim 1, wherein, The pseudo-label generation unit generates the comprehensive identification information based on the identification information of each attribute output from the multiple classifiers and the corresponding weight coefficients established with each classifier and attribute.

3. The image recognition auxiliary device as described in claim 1, wherein, The pseudo-label generation unit sets the frequency of occurrence of a characteristic value related to the attribute of the image in the image.

4. The image recognition auxiliary device as described in claim 1, wherein, The pseudo-label generation unit sets information representing the probability of the classifier recognizing the attribute for the characteristic values ​​related to the attributes of the image.

5. The image recognition auxiliary device as described in claim 1, wherein, It also includes a storage unit that stores the acquired image, the pseudo-tag of the image, and the new tag of the image in a corresponding manner.

6. The image recognition auxiliary device as described in claim 1, wherein, The images above are synthetic aperture radar (SAR) images or computer graphics (CG) images.

7. The image recognition auxiliary device as described in claim 1, wherein, It also includes a display unit that displays the identification information of the above-mentioned attributes and, based on the user's specification, sends the specified operation information to the terminal that has established a corresponding specification for the above-mentioned attributes.

8. The image recognition auxiliary device as described in claim 7, wherein, The aforementioned display unit displays identification information representing the attribute of a specified value or range for the aforementioned combined identification information related to the aforementioned pseudo-label.

9. The image recognition auxiliary device as described in claim 7, wherein, The aforementioned display unit displays information about the new labels corresponding to the aforementioned attributes, and accepts changes to the information of the new labels from the user.

10. The image recognition auxiliary device as claimed in claim 1, wherein, It also includes an image recognition model generation unit, which learns the relationship between multiple acquired images and newly generated labels corresponding to the multiple images, and generates a learned model that outputs the recognition information of each attribute of the input image.

11. An image recognition-assisted method, wherein, The information processing device performs the following: Image input processing to acquire images; The pseudo-label generation process involves recognizing the acquired image using each of several types of image recognition models and outputting recognition information. Based on the output recognition information, pseudo-labels representing the attributes of the acquired image are generated. The new tag generation process generates new tags based on the pseudo tags mentioned above. In the above pseudo-label generation process, the acquired image is input to multiple classifiers respectively. The multiple classifiers output recognition information for each attribute of the input image based on characteristic values ​​related to the attributes of the input image, and the pseudo-label generation process generates comprehensive recognition information for each attribute of the acquired image based on the recognition information for each attribute output from the multiple classifiers. Based on this comprehensive recognition information, pseudo-labels are generated. In the above new label generation process, the correctness of each attribute of the pseudo-label is calculated. Based on the calculated correctness, new labels are generated for each attribute of the obtained image. As the aforementioned classifiers, at least a low-frequency attribute model and a high-frequency attribute model are included. The low-frequency attribute model is a learning model that sets the learning weight of attributes with a probability of being present in the image below the first threshold (i.e., low-frequency attributes) to be higher than the learning weight of other attributes. The high-frequency attribute model is a learning model that sets the learning weight of attributes with a probability of being present in the image above the second threshold (which is greater than the first threshold) to be higher than the learning weight of other attributes. In the above pseudo-label generation process, based on the reliability of the probability of each attribute being present as output from the above classifiers respectively, which are input to the above classifiers respectively, and the weight coefficients corresponding to the occurrence frequency of each attribute established with each classifier and each attribute, the total value of the reliability of each attribute of the obtained image is generated.

12. A recording medium that is computer-readable and contains image recognition assistance program, wherein, The aforementioned image recognition assistance program is executed by the computer of the information processing device: Image input processing to acquire images; The pseudo-label generation process involves recognizing the acquired image using each of several types of image recognition models and outputting recognition information. Based on the output recognition information, pseudo-labels representing the attributes of the acquired image are generated. The new tag generation process generates new tags based on the pseudo tags mentioned above. In the above pseudo-label generation process, the acquired image is input to multiple classifiers respectively. The multiple classifiers output recognition information for each attribute of the input image based on characteristic values ​​related to the attributes of the input image, and the pseudo-label generation process generates comprehensive recognition information for each attribute of the acquired image based on the recognition information for each attribute output from the multiple classifiers. Based on this comprehensive recognition information, pseudo-labels are generated. In the above new label generation process, the correctness of each attribute of the pseudo-label is calculated. Based on the calculated correctness, new labels are generated for each attribute of the obtained image. As the aforementioned classifiers, at least a low-frequency attribute model and a high-frequency attribute model are included. The low-frequency attribute model is a learning model that sets the learning weight of attributes with a probability of being present in the image below the first threshold (i.e., low-frequency attributes) to be higher than the learning weight of other attributes. The high-frequency attribute model is a learning model that sets the learning weight of attributes with a probability of being present in the image above the second threshold (which is greater than the first threshold) to be higher than the learning weight of other attributes. In the above pseudo-label generation process, based on the reliability of the probability of each attribute being present as output from the above classifiers respectively, which are input to the above classifiers respectively, and the weight coefficients corresponding to the occurrence frequency of each attribute established with each classifier and each attribute, the total value of the reliability of each attribute of the obtained image is generated.

Images