Image recognition support device and image recognition support method

Abstract

To enable detection of a state / situation of an object included in an image.SOLUTION: An image recognition support apparatus 100 according to the present invention has an image acquiring unit 111 for acquiring an image, an image recognition unit 112 for detecting an object included in an image by using an object detection model 121, and a detection result processing unit 113 for generating one or more expansion image queries indicating partial images of the image including the object to set, as a detected object and an attribute detail label of the object, combination of highest similarity calculated by using an image language model 123 which learned relationship between an image and an attribute having a state / a situation among combinations of the expansion image queries and expansion language queries indicating one or more language labels.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image recognition support device and an image recognition support method for assigning attributes to an object included in an image.

Background Art

[0002] Aerial images and satellite images are powerful means for remotely grasping the situation on the ground. For example, the disaster situation can be grasped from aerial images and satellite images of the disaster site. As a method for displaying images (aerial images) transmitted from a plurality of aircraft, there is an information display method described in Patent Document 1. This display method aims to optimize the display of information including a plurality of images, generates a display area for an image transmitted from each aircraft, and displays the image from the aircraft selected by the user as the main image in the display area. In addition, an image in which a predetermined object is detected is displayed as the main image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 does not provide a detailed description of the display method of the detected object. For example, if the state / situation (e.g., isolated, flooded) of a recognition target (e.g., a person or a house) included in a photographed image of a disaster site can be detected, it is possible to support the rescue work of the disaster victims.

[0005] To achieve detection, it is necessary to prepare a classifier (machine learning model) that recognizes and classifies the objects to be recognized contained in an image. To create such a classifier, it is necessary to prepare pairs of images of the objects to be recognized and labels that represent those objects (ground truth labels) as training data, and train the classifier to learn these patterns. By using such a classifier, it is possible to detect the objects to be recognized in an image and assign labels to them.

[0006] After detecting the object to be recognized, it is necessary to further detect the detailed state of the object (e.g., flooded, collapsed, on fire, etc.) in order to understand its condition / situation. Furthermore, it is desirable to change the display when the appearance of the object changes within the overall image. For example, when the image changes from a micro view to a macro view, it is desirable to switch the display from "person" to "crowd" or to add the label "dangerous condition".

[0007] However, it is difficult to prepare correct labels that indicate the state / condition of an object as described above. Even if labels for general attributes of the object to be recognized (e.g., "building," "person," "car") can be prepared as training data, it is difficult to prepare all the labels that describe the object in detail (e.g., "broken," "flooded," "fallen," "buried in mud") as training data in advance. Furthermore, there is a desire to eliminate the need for manual setting of conditions that change the display according to changes in how the object appears, such as the size and number of objects to be recognized in the image.

[0008] This invention was made in view of the above background, and aims to provide an image recognition support device and an image recognition support method for detecting the state / condition of objects contained in an image. [Means for solving the problem]

[0009] To solve the above-mentioned problems, the image recognition support device according to the present invention includes an image acquisition unit that acquires an image and an object detection model that detects objects contained in the image.In addition, an attribute classification model is used to obtain attributes that indicate the state / condition of the object, and a linguistic label indicating the attributes of the object is detected. The image recognition unit generates one or more extended image queries that indicate a partial image of the image containing the object, The language label indicating the attribute, the language label indicating a synonym of the language label indicating the attribute, and the attribute detail label which is the language label indicating the state / condition of the object are obtained and used as an extended language query. The system includes a detection result processing unit that, among combinations of augmented image queries and augmented language queries indicating one or more language labels, obtains combinations with high similarity calculated using an image language model that has learned the relationship between an image and attributes including state / situation, and uses the object indicated by the augmented image query of that combination as the detected object, and the augmented language query of that combination as the attribute detail label of that object. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an image recognition support device and an image recognition support method for detecting the state / condition of objects contained in an image. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments. [Brief explanation of the drawing]

[0011] [Figure 1] This is a functional block diagram of the image recognition support device according to this embodiment. [Figure 2] This is a data structure diagram of the image database according to this embodiment. [Figure 3] This is a data configuration diagram of the display switching label according to this embodiment. [Figure 4] This is a data structure diagram of the attribute detail label according to this embodiment. [Figure 5] This figure illustrates the process for obtaining attribute detail labels according to this embodiment. [Figure 6] This is a flowchart of the image recognition support process according to this embodiment. [Figure 7] This is a flowchart of the attribute detail label display process according to this embodiment. [Figure 8] This is a flowchart of the display switching label display process according to this embodiment. [Figure 9] This is a screen configuration diagram of the image recognition result display screen according to this embodiment. [Figure 10]This is a screen configuration diagram of an extended query display screen according to the present embodiment. [Figure 11] This is a diagram showing an image recognition result display screen in which an area according to the present embodiment is specified. [Figure 12] This is a diagram showing an image recognition result display screen for displaying the recognition result of a group of objects according to the present embodiment.

Mode for Carrying Out the Invention

[0012] An image recognition support apparatus in a mode (embodiment) for carrying out the present invention will be described below. The image recognition support apparatus acquires, for example, an image of a disaster site taken by a camera mounted on a drone, and detects objects such as people and houses shown in the image using an object detection model. Next, the image recognition support apparatus detects an attribute of the detected object (for example, the attribute of "collapsed" for a house) using an attribute classification model.

[0013] Subsequently, the image recognition support apparatus generates a partial image (extended image query) including an object and language labels (extended language query) such as synonyms of attributes, set attributes, and words specified by the user. The image recognition support apparatus acquires a combination with a high similarity calculated by the image language model among the combinations of the extended image query and the extended language query, and uses the extended language query of the combination as an attribute detailed label of the object. The image recognition support apparatus assigns and displays an attribute detailed label to the object. Note that the language label is a label indicated by a language (text). For example, the attribute detailed label is a language label that is text indicating the attribute of the object. Also, the object detection label described later is a language label indicating the general name of the object. The language label may be simply referred to as a label.

[0014] In addition, the image recognition support apparatus acquires a language label with a high similarity to the entire image by the image language model for a label (language label, text) that is an attribute of a plurality of objects rather than an attribute of an individual object, and assigns and displays it to the image as a display switching label. Examples of language labels that can be candidates for the display switching label include "crowd" and "dangerous (state)", which are registered in advance.

[0015] According to such an image recognition support device, an attribute detailed label indicating the state / situation of an object shown in an image can be assigned to and displayed for the object. Consequently, the user of the image recognition support device can quickly and accurately grasp the situation of an object present at the scene shown in the image (for example, a house at a disaster scene).

[0016] To prepare a machine learning model for obtaining an attribute detailed label for an object, a large amount of training data in the form of pairs of objects and attribute detailed labels is required, and it is difficult to prepare such training data. By generating an extended image query and an extended language query and calculating the similarity using an image language model, an attribute detailed label for an object can be obtained without preparing a large amount of training data. In addition, by the image recognition support device displaying a display switching label, the situation of the entire scene shown in the image can be quickly and accurately grasped.

[0017] ≪Configuration of Image Recognition Support Device≫ FIG. 1 is a functional block diagram of an image recognition support device 100 according to the present embodiment. The image recognition support device 100 is a computer and includes a control unit 110, a storage unit 120, and an input / output unit 180. User interface devices such as a display, a keyboard, and a mouse are connected to the input / output unit 180. The input / output unit 180 may include a communication device and be capable of transmitting and receiving data (for example, images from a surveillance camera or a drone) with other devices. Further, a media drive may be connected to the input / output unit 180, and data exchange using a recording medium may be possible.

[0018] The memory unit 120 is comprised of memory devices such as ROM (Read Only Memory), RAM (Random Access Memory), and SSD (Solid State Drive). The memory unit 120 stores an image database 130, display switching labels 140, attribute detail labels 150, an object detection model 121, an attribute classification model 122, an image language model 123, and a program 128. The program 128 contains a description of the processing procedure in the image recognition support processing (see Figure 6) described later.

[0019] ≪Memory Section: Image Database≫ Figure 2 is a data structure diagram of the image database 130 according to this embodiment. The image database 130 is, for example, tabular data, where one row (record) represents one image. A record includes columns (attributes) of identification information (labeled "ID" in Figure 2), object detection label, attribute label, display switching label, attribute detail label, metadata, and image data.

[0020] The identification information indicates the identification information of the image. The object detection label contains a list of tuples that include the location, label, and confidence level of an object in the image detected by the image recognition unit 112 (described below). The location indicates the region in the image where the object is located. The label is a general name / language label of the object, such as "house" or "person". The confidence level is the degree / probability that the object in the image region indicated by the location is the object indicated by the label. In the following explanation, the labels included in the object detection label may be referred to simply as the object detection label.

[0021] The attribute labels include a list of pairs, each containing the location of an object in the image and a label indicating the attribute of that object as detected by the image recognition unit 112 (described later). The location is the object's position (region) and corresponds to the location included in the object detection label. The attributes are the object's attributes; for example, a house might have attributes such as "collapsed" or "flooded."

[0022] The display switching label includes the label (display switching label) contained in display switching label 140 (see Figure 3 below) and a list of pairs including the similarity between the image and the label calculated using the image language model 123 below. The attribute detail label includes a list of tuples containing the location of the object in the image, the label included in attribute detail label 150 (see Figure 4 below), and the similarity between the label and the image of the object (the image corresponding to the object's location) calculated using the image language model 123. The label may also include the attribute of the object detected by the image recognition unit 112, or synonyms for the attribute and attribute detail label (see extended language query below). Metadata refers to the metadata of an image, and includes information such as the date and location where the image was taken. Image data is the image itself (data).

[0023] ≪Storage Unit: Display Switching Label≫ Figure 3 is a data configuration diagram of the display switching label 140 according to this embodiment. The display switching label 140 includes one or more labels (display switching labels). The display switching labels are registered (added / deleted) by the user or administrator of the image recognition support device 100. Display switching labels are labels for multiple objects in an image, either as a whole or as a sub-image, rather than for individual objects. Examples of display switching labels include labels that indicate the names of multiple people in an image, such as "crowd," "gathering," or "group." Other examples include labels that indicate the state, situation, or attribute of multiple people, such as "dangerous," "orderly," or "excited."

[0024] ≪Memory Section: Attribute Details Label≫ Figure 4 is a data structure diagram of the attribute detail label 150 according to this embodiment. The attribute detail label 150 includes one or more labels (attribute detail labels). Attribute detail labels are registered (added / deleted) by the user or administrator of the image recognition support device 100. Attribute detail labels are labels for objects. Examples of attribute detail labels for a house include "normal," "flooded," and "collapsed."

[0025] ≪Memory Unit: Object Detection Model≫ Returning to Figure 1, let's continue the explanation of the memory unit 120. The object detection model 121 is a machine learning model used by the image recognition unit 112 (described later) to detect objects in an image and obtain the object's location (region) and a general name label (such as "person" or "house"). The output of the process may include confidence in addition to the location and label. The image recognition unit 112 stores the output in the object detection label of the image database 130 (see Figure 2). Therefore, the label is also the object detection label.

[0026] ≪Memory Unit: Attribute Classification Model≫ The attribute classification model 122 is a machine learning model used to acquire the attributes of objects detected by the image recognition unit 112, which will be described later. The output of the process may include confidence scores in addition to attributes. The image recognition unit 112 stores the output in the attribute labels of the image database 130 (see Figure 2).

[0027] Examples of objects to which attributes are assigned include "damaged building," "flooded building," "fallen person," and "car buried in mud." However, it is difficult to prepare training data for the attribute classification model 122 in a way that allows for the acquisition of such attributes. Therefore, the attributes obtained using the attribute classification model 122 do not necessarily accurately represent the state / condition of the object. In addition to the attributes obtained using the attribute classification model 122, the image recognition support device 100 uses the image language model 123 described later to acquire appropriate labels indicating the state / condition of an object from among the synonyms of the attribute and the labels included in the attribute detail label 150 (see Figure 4).

[0028] Furthermore, deep learning models are one type of machine learning model, such as the object detection model 121 and the attribute classification model 122. Examples of such models include convolutional neural networks (CNNs) and vision transformers, which are composed of networks with multiple layers.

[0029] ≪Memory Unit: Image Language Model≫ The image language model 123 is a machine learning model that shows the relationship between images and language (text). Examples of text include "photo of a dog," "cute cat," and "dog lying down." While the object detection model 121 learns the relationship between images and labels (general names such as "person" and "house"), the image language model 123 learns the relationship between images and attributes, including state / situation. By using the image language model 123, it is possible to calculate the similarity between an image and its attributes, and attributes with a high similarity to an image can be considered to represent the attributes of that image. CLIP (Contrastive Language-Image Pre-training) is an example of the image language model 123.

[0030] ≪Control Unit≫ Following the description of the storage unit 120, the control unit 110 will be described. The control unit 110 includes a CPU (Central Processing Unit) and comprises an image acquisition unit 111, an image recognition unit 112, a detection result processing unit 113, a clustering unit 114, and a display control unit 115.

[0031] ≪Control Unit: Image Acquisition Unit≫ The image acquisition unit 111 acquires images via the input / output unit 180 and stores them in the image data of the image database 130 (see Figure 2). If metadata is attached to the image, it is also stored in the metadata of the image database 130. The images may be, for example, images taken by a person-held camera or a device fixed to the ground. Alternatively, the images may be taken by a camera mounted on a vehicle moving on the ground, or by a camera mounted on a drone, aircraft, satellite, etc. The camera may be connected to the image recognition support device 100 via a network, or it may be connected directly. The images may also be previously taken images / videos stored on a recording medium.

[0032] As described above, the image recognition support device 100 includes an image acquisition unit 111 that acquires images.

[0033] ≪Control Unit: Image Recognition Unit≫ The image recognition unit 112 uses the object detection model 121 to detect objects in the acquired image, obtain their location (region), label (object detection label), and confidence level, and store them in the object detection label of the image database 130 (see Figure 2). Subsequently, the image recognition unit 112 uses the attribute classification model 122 to obtain the attributes (label, language label) of the object (image of the object indicated by its location) and store them in the attribute label of the image database 130.

[0034] As described above, the image recognition support device 100 includes an image recognition unit 112 that detects objects contained in an image using an object detection model 121. The image recognition unit 112 uses the attribute classification model 122 to detect linguistic labels that indicate the attributes of an object.

[0035] ≪Control Unit: Detection Result Processing Unit≫ The detection result processing unit 113 uses the image language model 123 to obtain the image display switching label and the attribute detail label of the detected object. First, let's explain the display switching labels. The detection result processing unit 113 calculates the similarity between the image acquired by the image acquisition unit 111 and each of the display switching labels included in the display switching label 140 (see Figure 3) using the image language model 123, and stores it in the display switching labels of the image database 130 (see Figure 2). Furthermore, if the average value of the top similarities is greater than or equal to a predetermined value, the detection result processing unit 113 notifies the display control unit 115 (described later) to display the display switching label corresponding to that similarity. For more information on average values, please refer to the explanation of average values ​​in the attribute detail labels.

[0036] Next, attribute detail labels will be explained. Figure 5 is a diagram illustrating the acquisition process of attribute detail labels according to this embodiment. The original image 311 is an image of an object detected by the image recognition unit 112. The detection result processing unit 113 generates extended images 312 to 314, which include the original image 311, overlap with the original image 311, or are included in the original image 311. Although Figure 5 shows three extended images 312 to 314, there are not necessarily three. Extended images 312 to 314 that include the original image 311 are also referred to as extended image queries 310.

[0037] The original text 321 is the attribute of the object detected by the image recognition unit 112. The detection result processing unit 113 obtains synonyms of the original text 321, extensions / additions using templates, and labels included in the attribute detail label 150 to create extended texts 322-324. Extended texts 322-324 may also be attributes (language labels, text) specified by the user.

[0038] A template is, for example, a template that adds "photo of ~" to an object such as a house if it has the attribute "collapsed," such as "photo of a collapsed house," or a template that adds "state of ~." Although Figure 5 shows three extended texts, there are not necessarily three. The extended texts 322-324, including the original text 321, are also referred to as the extended language query 320.

[0039] Next, the detection result processing unit 113 generates combinations of extended image queries 310 (original image 311, extended images 312-314) and extended language queries 320 (original text 321, extended text 322-324). Next, the detection result processing unit 113 calculates the similarity of each combination using the image language model 123 and calculates the average of the similarities in a predetermined number / predetermined ratio from the combinations with high similarity. The average can be an arithmetic mean or any other mean such as a geometric mean. If the average similarity is greater than or equal to a predetermined value, the detection result processing unit 113 stores the extended image queries and extended language queries included in the combination, as well as the similarity of the combination, in the attribute detail label of the image database 130 (see Figure 2), and also notifies the display control unit 115, which will be described later. The display control unit 115 displays the extended image queries and extended language queries (see Figure 9). In Figure 5, the similarity is indicated by the thickness of the line connecting the extended image queries and extended language queries. For example, the line connecting the extended image 314 and the extended text 322 is the thickest, indicating that the similarity between the extended image 314 and the extended text 322 is the highest.

[0040] As described above, the image recognition support device 100 generates one or more extended image queries indicating a partial image of an image containing an object, and among the combinations of the extended image queries and extended language queries indicating one or more language labels, it obtains a combination with a high similarity calculated using an image language model 123 that has learned the relationship between the image and attributes including state / situation, and includes a detection result processing unit 113 that identifies the object indicated by the extended image query of the combination as the detected object, and the extended language query of the combination as the attribute detail label of the object. The detection result processing unit 113 calculates the display switching label with the highest similarity among the combinations of an image and one or more display switching labels (see display switching label 140) that indicate language labels, using the image language model 123, and uses that label as the display switching label for the image.

[0041] Extended language queries include pre-configured language labels (see attribute detail label 150) that indicate the attributes of an object. An extended language query includes at least one of the following: a language label indicating an attribute (of an object), and a language label indicating a synonym for that attribute. Extended language queries include language labels specified (by the user).

[0042] ≪Control Unit: Clustering Unit, Display Control Unit≫ The clustering unit 114 performs a clustering process that groups objects within a specified region of the image into groups based on their proximity. The display control unit 115 displays the processing results of the image recognition unit 112 and the detection result processing unit 113 on a display connected to the input / output unit 180 as an image recognition result display screen 400 (see Figure 9 below) and an extended query display screen 430 (see Figure 10).

[0043] As described above, the image recognition support device 100 includes a clustering unit 114 that performs clustering processing on objects contained in a specified area of ​​an image, dividing objects that are close in distance into multiple groups.

[0044] Image Recognition Support Processing Figure 6 is a flowchart of the image recognition support process according to this embodiment. In step S11, the image acquisition unit 111 starts the process of repeating steps S12 to S19. In step S12, the image acquisition unit 111 acquires an image and stores it in the image database 130 (see Figure 2; labeled as image DB (database) in Figure 6).

[0045] In step S13, the image recognition unit 112 uses the object detection model 121 to detect objects in the image. In step S14, the image recognition unit 112 uses the attribute classification model 122 to detect (acquire) the attributes of the object detected in step S13. The attribute detail label display process in step S15 (see Figure 7 below) and the display switching label display process in step S16 (see Figure 8 below) are processed concurrently. Details of steps S15 and S16 will be described later with reference to Figures 7 and 8.

[0046] In step S17, the detection result processing unit 113 determines whether or not to save the extended language queries that are not present in the display switching label 140 (see Figure 3) or the attribute detail label 150 (see Figure 4). If the detection result processing unit 113 decides to save them (step S17 → YES), it proceeds to step S18; if it decides not to save them (step S17 → NO), it proceeds to step S19. The procedure for making this decision will be explained in conjunction with step S18, which will be described later.

[0047] In step S18, the detection result processing unit 113 adds the extended language query to the display switching label 140 or the attribute detail label 150 and saves it. For example, if the extended language query is an attribute acquired by the image recognition unit 112, or a synonym of said attribute, and the average of the top similarity values ​​between the extended language query and the extended image query calculated using the image language model 123 is greater than or equal to a predetermined value (see steps S33 to S35 in Figure 7), the detection result processing unit 113 saves it to the attribute detail label 150. Also, if the similarity between the label specified by the user and the image calculated using the image language model 123 is greater than or equal to a predetermined value, the detection result processing unit 113 saves it to the display switching label 140.

[0048] In step S19, the image acquisition unit 111 determines whether the process is complete or not. If it is complete (step S19 → YES), it terminates; otherwise (step S19 → NO), it returns to step S12. The image acquisition unit 111 determines the process is complete, for example, if there is no image input or if the termination menu is selected on the image recognition result display screen 400.

[0049] ≪Attribute detail label display processing≫ Figure 7 shows the attribute detail label display process according to this embodiment (Figure 6 This is a flowchart of step S15 described above. In step S31, the detection result processing unit 113 starts repeating steps S32 to S36 for each object detected by the image recognition unit 112 in step S13 (see Figure 6). In step S32, the detection result processing unit 113 generates an extended image query and an extended language query (see Figure 5).

[0050] In step S33, the detection result processing unit 113 calculates the similarity for each combination of an extended image query and an extended language query using the image language model 123. In step S34, the detection result processing unit 113 calculates the average value of the similarities that are among the top in terms of a predetermined number / predetermined ratio from the similarities calculated in step S33. In step S35, the detection result processing unit 113 proceeds to step S36 if the average value calculated in step S34 is greater than or equal to a predetermined value (step S35 → YES), and returns to step S32 to process the object if it is less than the predetermined value (step S35 → NO).

[0051] In step S36, the detection result processing unit 113 instructs the display control unit 115 to display the location of the object corresponding to the highest similarity (extended image query) and the extended language query, which is an attribute detail label (see Figure 9 below).

[0052] <Display switching label display processing> Figure 8 shows the display switching label display process according to this embodiment (Figure 6 This is a flowchart of step S16 described above. In step S41, the detection result processing unit 113 calculates the similarity between the image acquired in step S12 (see Figure 6) and each label (display switching label) included in the display switching label 140 (see Figure 3) using the image language model 123. In step S42, the detection result processing unit 113 calculates the average value of the similarities that are among the top in terms of a predetermined number / predetermined ratio from the similarities calculated in step S41.

[0053] In step S43, the detection result processing unit 113 proceeds to step S44 if the average value calculated in step S42 is greater than or equal to a predetermined value (step S43 → YES), and terminates the process if it is less than the predetermined value (step S43 → NO). In step S44, the detection result processing unit 113 instructs the display control unit 115 to display the display switching label corresponding to the higher similarity in step S42 (see Figure 9 below).

[0054] ≪Image recognition result display screen≫ Figure 9 is a screen configuration diagram of the image recognition result display screen 400 according to this embodiment. The upper right area 421 of the image recognition result display screen 400 displays the original image input to the image recognition support device 100. The middle right area 422 of the image recognition result display screen 400 displays a map showing the shooting location of the original image (see black circle).

[0055] In the lower right area 423 of the image recognition result display screen 400, images similar to the original image are displayed. Similar images are images that have similar or identical attributes to the images in area 425 described below, or to objects depicted in the original image. Similar images may also be images taken at nearby locations, or images that have similarities other than attributes (for example, the distribution of pixel colors). By referring to these similar images, users can deepen their understanding of the image's attributes.

[0056] In the pull-down menu 424, you can select options such as "Detection Results" and "Extended Query". In the image recognition results display screen 400 shown in Figure 9, "Detection Results" is selected, and the detection results of objects in the image (original image) are displayed in the left area 425 of the image recognition results display screen 400.

[0057] A display switching label 427 is displayed above region 425. This display switching label is displayed in accordance with step S44 of the display switching label display process (see Figure 8), and is one or more display switching labels that have a high similarity to the original image calculated using the image language model 123 (see steps S42 and S43).

[0058] For each of the detected objects 412, 415, and 418, regions 413, 416, and 419 (location, extended image query) and attribute detail labels 411, 414, and 417 (labeled "####" in Figure 9) representing the object 412, 415, and 418 are displayed. These regions 413, 416, and 419 and attribute detail labels 411, 414, and 417 are the location and attribute detail labels of the object with the highest similarity (see step S36 in Figure 7). Note that the attribute detail labels 411, 414, and 417 may also include the similarity value. This display allows users of the image recognition support device 100 to easily grasp the state / condition of the image and the objects 412, 415, and 418 depicted in the image.

[0059] As described above, the image recognition support device 100 includes a display control unit 115 that outputs an image recognition result display screen 400 which includes an image (see region 425) showing the combination of an extended image query and an extended language query that has the highest similarity among the combinations of extended image queries and extended language queries relating to an object.

[0060] ≪Extended Query Display Screen≫ When "Extended Query" is selected in the pull-down menu 424, the image recognition result display screen 400 switches to the extended query display screen 430 (see Figure 10 below). Figure 10 is a screen configuration diagram of the extended query display screen 430 according to this embodiment. When "Extended Query" is selected in the pull-down menu 431, the extended image query and extended language query for the objects in the image are displayed in the left area 432 of the extended query display screen 430. Below, the extended image query and extended language query for the object 418 in the lower center (see Figure 9) will be explained.

[0061] Regions 441, 443, and 445 (locations) represent extended image queries. Labels 442, 444, and 446 represent the similarity to the extended language queries (attribute detail labels, language labels) that had the highest similarity for regions 441, 443, and 445, respectively (see step S34 in Figure 7). This display allows users of the image recognition support device 100 to understand which parts of the image (original image) the image recognition support device 100 is recognizing and how it is recognizing the image and the objects 412, 415, and 418 (see Figure 9) contained in the image.

[0062] As described above, the image recognition support device 100 includes a display control unit 115 that outputs an extended query display screen 430 which includes an extended image query, an extended language query, and an image (see region 432) showing the similarity between the extended image query and the extended language query calculated using the image language model 123.

[0063] Label editing operations Users of the image recognition support device 100 can edit the display switching label 140 (see Figure 3) and the attribute detail label 150 (see Figure 4) on the image recognition result display screen 400 or the extended query display screen 430. More specifically, for example, when "Edit Label" is selected from the pull-down menus 424 and 431, the detection result processing unit 113 displays the editing screen (not shown) for the display switching label 140 and the attribute detail label 150. When the user is instructed to finish editing, the detection result processing unit 113 stores the editing results in the display switching label 140 and the attribute detail label 150. By using these editing operations, users can disable unnecessary display toggle labels and attribute detail labels, or add display toggle labels and attribute detail labels that they want to be displayed when detected.

[0064] <<Summarization Operation>> A user of the image recognition support device 100 can obtain a summary result of objects in an image by specifying an area of ​​the image on the image recognition result display screen 400. Figure 11 shows the image recognition result display screen 450 with area 451 specified according to this embodiment. On the image recognition result display screen 450, 14 objects (hatched circles) are detected, and the area and attribute detail label are displayed for each object. Now, suppose the user specifies area 451 and instructs the system to perform a summary.

[0065] The clustering unit 114 then divides the detected objects into groups based on distance. Next, the objects included in each group are treated as a single object, and the processes in steps S14 to S19 (see Figure 6) are executed. Furthermore, if the position of the object detected in the next image during the iterative process in Figure 6 (see step S13) is close to the position of the object when region 451 was specified (for example, the difference in position is less than or equal to a predetermined value), the processes from grouping by the clustering unit 114 onward are repeated.

[0066] Figure 12 shows an image recognition result display screen 460 that displays the recognition results of object groups according to this embodiment. The 14 objects are divided into groups of 5 objects, 5 objects, and 4 objects from top to bottom, and an area indicating the position of each group and attribute detail labels for the group are displayed. The process of detecting (acquiring) the attribute detail labels is performed by the detection result processing unit 113 (see attribute detail label display process described in Figure 7).

[0067] By utilizing this type of summarization operation, users can, for example, in an image containing multiple people, display several people (groups of people) who are close together as a crowd, or obtain information about the state / condition of the crowd.

[0068] As described above, the detection result processing unit 113 treats a group of objects as a single object and calculates the group and its attribute detail labels.

[0069] Features of the Image Recognition Support Device The image recognition support device 100 detects an object using the object detection model 121, and then detects the attributes of the detected object using the attribute classification model 122. Subsequently, the image recognition support device 100 generates an extended image query and an extended language query, and obtains the combination of the extended image query and extended language query that has the highest similarity calculated by the image language model 123. The extended language query of that combination is then used as the attribute detail label for the object, and the attribute detail label is attached (superimposed) on the object and displayed.

[0070] The image recognition support device 100 can calculate a more accurate similarity than calculating similarity using the original image and language label by calculating the similarity of combinations of one or more extended image queries and one or more extended language queries. This is because a rectangular image of the object itself is not always the best image to represent the object, and the background surrounding the object may contribute as information indicating the object or its state / condition. Similarly, the language label specified in advance by the user is not always the best label (text) to indicate the state of the object, and a synonym or a label converted using a template may be more appropriate.

[0071] When detecting (acquiring and describing) the attributes of objects in an image using the image language model 123, using extended image queries and extended language queries allows for the acquisition of more accurate and detailed states / situations. Consequently, the need for manual label correction is reduced, making image recognition simpler and faster.

[0072] The image recognition support device 100 uses mechanisms such as image augmentation, synonym search, and template-based addition when creating extended queries (extended image queries and extended language queries) for the original image and language labels. This allows it to create extended queries that the user might not have thought of. Furthermore, the created extended queries can be filtered out by excluding pairs of images and language labels with low similarity using the image language model 123, thereby eliminating inappropriate images and language labels.

[0073] By registering labels for micro and macro perspectives in the display switching label 140 and attribute detail label 150, the image recognition support device 100 will switch the display in accordance with screen changes. For example, if labels such as "person" and "crowd" are registered in the display switching label 140 (see Figure 3), in video footage where the view changes significantly, such as images taken by a drone, the display switching label 427 (see Figure 9) will switch depending on whether the image is a micro perspective image mainly showing "people" or a macro perspective image mainly showing "crowds". This makes it possible to automatically perform a simple and intuitive screen display.

[0074] <<Variation Example: Detailed Description of Attributes of a Specified Object>> In the summarization operation, the image recognition support device 100 groups one or more objects in the area specified by the user (see area 451 in Figure 11) and displays attribute detail labels (see Figure 12). The image recognition support device 100 may also display attribute detail labels for a partial image containing one or more objects specified by the user. More specifically, the image recognition support device 100 (detection result processing unit 113) generates an extended image query 310 with the specified partial image as the original image 311 (see Figure 5). The image recognition support device 100 also generates an extended language query 320 with the attributes detected from the original image 311 using the attribute classification model 122 as the original text 321. Subsequently, the image recognition support device 100 can display attribute detail labels by executing steps S33 to S36 (see Figure 7). In this way, even if object detection by the object detection model 121 fails, attribute detail labels can still be obtained.

[0075] Furthermore, the system may allow users to specify language labels for one or more designated objects or groups of objects. The image recognition support device 100 performs attribute detail label display processing (see Figure 7) for the designated objects / groups of objects and language labels, and displays the attribute detail label if the average similarity value is greater than or equal to a predetermined value. In this way, users can determine whether the labels they have noticed are appropriate or not.

[0076] As explained above, an extended image query includes partial images within the specified image. The detection result processing unit 113 treats the specified multiple objects as a single object and calculates an attribute detail label for that single object.

[0077] ≪Extreme transformation: Image≫ The images handled by the image recognition support device 100 are not limited to monochrome or RGB images; they may also be, for example, infrared images or CG (Computer Graphics) images. This allows for image recognition support even in cases where satellite images and aerial images, which often use thermal images or composite images, are utilized. Furthermore, even in cases where biases in the frequency of appearance of attributes are likely to occur, such as when assessing disaster situations in aerial images, and where it is necessary to assign labels that are rarely used in training data, highly accurate detailed information can be assigned to attributes in the image with minimal effort to add training data or perform manual labeling.

[0078] As explained above, the images can be monochrome images, color images, infrared images, or computer graphics images.

[0079] <<Variation: Extended Image Query>> The augmented image query in the above-described embodiment includes a set of images that include the region surrounding the object. The augmented image query may also be an image of the object that has undergone various image transformation processes. Examples of image transformations include color transformation, super-resolution, affine transformation, text removal, and noise reduction. Furthermore, the process of cropping the detection target region and the image transformation process may be performed simultaneously.

[0080] ≪Variation: Work support function≫ The image recognition support device 100 may also include a function to send work instructions (such as text or voice) to the photographer taking the image or to a worker at the shooting site. This allows for the execution of various tasks according to the image recognition status. For example, appropriate disaster relief and recovery can be carried out based on images depicting the disaster situation.

[0081] <<Example: Extended Query Display Screen>> Area 432 of the extended query display screen 430 (see Figure 10) displays extended image queries and extended language queries for objects in the image. Alternatively, the display format may be changed to show the similarity for all combinations of extended image queries and extended language queries (see Figure 5).

[0082] <<Other variations>> Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. For example, an alarm may be issued when a pre-specified label (e.g., "Danger" or "Abnormal") is detected among the labels included in the display switching label 140 or attribute detail label 150, or when the similarity exceeds a predetermined threshold.

[0083] Attribute detail labels and display toggle labels are displayed if the average similarity of the top labels is equal to or greater than a predetermined value (see steps S34 and S35 in Figure 7, and steps S42 and S43 in Figure 8). Attribute detail labels and display toggle labels may also be displayed if their similarity is equal to or greater than a predetermined value.

[0084] The present invention can take on various other embodiments, and furthermore, various modifications such as omissions and substitutions can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention as described herein, and are included in the scope of the invention and its equivalents as described in the claims. [Explanation of symbols]

[0085] 100 Image Recognition Support Device 111 Image acquisition unit 112 Image Recognition Unit 113 Detection Result Processing Unit 114 Clustering section 115 Display Control Unit 121 Object Detection Models 122 Attribute Classification Models 123 Image Language Model 128 Programs 130 Image Databases 140 Display Switching Labels 150 Attribute Details Labels 310 Extended Image Queries 320 Extended Language Queries 400 Image recognition result display screen 411,414,417 Attribute detail labels (extended language queries and language labels displayed on the screen) 412,415,418 Objects (Objects displayed on the screen) 413,416,419,441,443,445 Region (Region indicating the location of an object, extended image query displayed on the screen) 430 Extended Query Display Screen

Claims

1. Image acquisition unit that acquires images, An image recognition unit that uses an object detection model to detect objects contained in the image and uses an attribute classification model to obtain attributes indicating the state / condition of the object to detect a language label indicating the attribute of the object. Generate one or more extended image queries that show partial images of the image containing the object in question. The language label indicating the attribute, the language label indicating a synonym of the language label indicating the attribute, and the attribute detail label which is a language label indicating the state / condition of the object are obtained and used as an extended language query. Among the combinations of the augmented image query and augmented language queries that indicate one or more language labels, we obtain the combination with the highest similarity calculated using an image language model that has been trained on the relationship between the image and attributes including state / situation. The system comprises: an object detected by an extended image query of the combination; and a detection result processing unit that uses the extended language query of the combination as an attribute detail label for the object. Image recognition support device.

2. The aforementioned extended image query is: Includes a portion of the specified image The image recognition support device according to claim 1.

3. The display control unit further comprises an extended query display screen that outputs an extended query display screen including the extended image query, the extended language query, and an image showing the similarity between the extended image query and the extended language query calculated using the image language model. The image recognition support device according to claim 1.

4. The display control unit further comprises an image recognition result display screen that outputs an image showing an extended image query and an extended language query that represent the combination of extended image queries and extended language queries that has the highest similarity among the combinations of extended image queries and extended language queries relating to the aforementioned object. The image recognition support device according to claim 1.

5. The aforementioned detection result processing unit, Among the combinations of the aforementioned image and display switching labels indicating one or more language labels, the display switching label with the highest similarity calculated using the image language model is calculated as the display switching label for the said image. The image recognition support device according to claim 1.

6. The aforementioned detection result processing unit, Treat the specified multiple objects as a single object and calculate the attribute detail label for that single object. The image recognition support device according to claim 1.

7. The system further includes a clustering unit that performs clustering on the objects contained in a specified region within the aforementioned image to divide objects that are close together into multiple groups. The aforementioned detection result processing unit, Treat the group as a single object and calculate the group and its attribute detail labels. The image recognition support device according to claim 1.

8. The aforementioned image may be a monochrome image, a color image, an infrared image, or a computer graphics image. The image recognition support device according to claim 1.

9. The image recognition support device, Steps to acquire an image, The process involves using an object detection model to detect objects included in the image, and using an attribute classification model to obtain attributes indicating the state / condition of the objects to detect linguistic labels indicating the attributes of the objects. Generate one or more extended image queries that show partial images of the image containing the object in question. The language label indicating the attribute, the language label indicating a synonym of the language label indicating the attribute, and the attribute detail label which is a language label indicating the state / condition of the object are obtained and used as an extended language query. Among the combinations of the augmented image query and augmented language queries that indicate one or more language labels, we obtain the combination with the highest similarity calculated using an image language model that has been trained on the relationship between the image and attributes including state / situation. The steps are to identify the object indicated by the augmented image query for the given combination as the detected object, and to use the augmented language query for the given combination as the attribute detail label for that object. Image recognition support method.

Images