Information processing device and information processing method

Abstract

To provide a technique with which the basis for image similarity can be explained on a conceptual basis.SOLUTION: In an information processing device, a plurality of feature vectors (in a feature space) output from a learning model in response to an input of a plurality of input images to the learning model subjected to machine learning are acquired (step S21). Further, a plurality of hierarchical clusters are generated by performing hierarchical clustering processing on the plurality of feature vectors (step S22). A partial space or vector corresponding to a specific cluster among the plurality of clusters is then extracted as a concept of the specific cluster (concept expression) (step S23).SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device for machine learning (particularly, an information processing device for improving explainability regarding machine learning), and techniques related thereto. [Background technology]

[0002] In recent years, inference processing technology using machine learning such as deep learning has rapidly evolved.

[0003] However, due to the fact that the learning models in machine learning are extremely complex, there is a problem in that it is not always clear (it is not easy to explain) what basis the inference results obtained by the learning model are based on.

[0004] In particular, in situations where the inference result has a significant impact, there is a demand for improved explainability of the decision basis. For example, the technology described in Patent Document 1 is one of the technologies that meets this demand. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-33376 Summary of the Invention [Problem to be solved by the invention]

[0006] In the above-mentioned Patent Document 1, an area of ​​interest within an image area (an area of ​​interest in inference processing using a learning model) is visualized, and the area of ​​interest is recognized as an area used for inference. In other words, it is possible to recognize which area within the image was focused on to determine similarity.

[0007] However, with the technology of Patent Document 1, even if it is possible to grasp the region of interest (position of interest) within an image in the inference process, it is difficult to grasp what concept similarity influences the inference result. In other words, it is difficult to explain the basis for judgment in the inference process on a concept basis.

[0008] Therefore, an object of the present invention is to provide a technology that can explain the basis of image similarity on a concept basis. [Means for solving the problem]

[0009] In order to solve the above problem, an information processing device according to the present invention is configured to: Corresponding to each input of The output from the learning model Complex and a control unit that acquires a number of feature vectors, generates a number of hierarchical clusters by performing a hierarchical clustering process on the plurality of feature vectors, and extracts a subspace or vector corresponding to a specific cluster from the plurality of clusters as a concept of the specific cluster.

[0010] The control unit may extract a representative vector relating to the specific cluster as a concept of the specific cluster.

[0011] The control unit may generate a concept visualization image, which is a virtual input image corresponding to the representative vector and which visualizes the concept of the specific cluster, based on the representative vector and the learning model, and display the concept visualization image on a display unit.

[0012] The information processing device may further include a receiving unit that receives input of attribute information of the specific cluster.

[0013] When determining the similarity between the first image and the second image based on a first feature vector output from the learning model in response to an input of a first image to the learning model and a second feature vector output from the learning model in response to an input of a second image to the learning model, the control unit determines the similarity between the first image and the second image for at least one concept among a plurality of concepts respectively corresponding to the plurality of clusters. Judgment of The contribution to the

[0014] When the control unit determines the similarity between the first image and the second image based on a first feature vector output from the learning model in response to a first image input to the learning model and a second feature vector output from the learning model in response to a second image input to the learning model, the control unit may extract, from among a plurality of subspaces corresponding respectively to the plurality of clusters, a subspace that relatively reduces the distance between the projection vectors of the first and second feature vectors onto the orthogonal complementary space, or a vector spanning the subspace, as a concept that serves as the basis for determining that the first image and the second image are not similar to each other.

[0015] Furthermore, the information processing device according to the present invention is configured to input a plurality of input images to a machine-learned learning model. Corresponding to each input of The output from the learning model Complex and a control unit that acquires a number of feature vectors, generates a number of hierarchical clusters by performing a hierarchical clustering process on the plurality of feature vectors, and determines two or more input images corresponding to a specific cluster among the plurality of clusters as a group of images that represent the concept of the specific cluster.

[0016] Further, the information processing method according to the present invention includes: a) inputting a plurality of input images to a machine-learned learning model; Corresponding to each input of The output from the learning model Complexa) acquiring a number of feature vectors; b) generating a number of hierarchical clusters by performing a hierarchical clustering process on the plurality of feature vectors; and c) extracting a subspace or vector corresponding to a specific cluster from the plurality of clusters as a concept of the specific cluster. [Effects of the Invention]

[0017] According to the present invention, it is possible to explain the basis of image similarity on a concept basis. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a schematic diagram illustrating an image processing system. [Figure 2] FIG. 1 is a conceptual diagram illustrating an overview of processing in an image processing device. [Figure 3] 10 is a flowchart showing processing in the image processing device. [Figure 4] 10 is a flowchart illustrating a concept analysis process. [Figure 5] 10 is a flowchart illustrating a process for explaining the basis for similarity determination. [Figure 6] 10 is a flowchart illustrating a process for explaining the basis for determining dissimilarity. [Figure 7] FIG. 10 is a conceptual diagram showing the learning process in the first phase. [Figure 8] FIG. 10 is a diagram showing a feature space etc. in a state where learning has progressed. [Figure 9] FIG. 10 is a diagram illustrating the inference processing in the second phase. [Figure 10] FIG. 10 is a diagram illustrating an example of an inference processing result. [Figure 11] FIG. 10 is a diagram showing dendroids (tree diagrams) and the like related to the results of hierarchical clustering processing. [Figure 12] FIG. 10 is a diagram illustrating a hierarchical relationship around a specific cluster. [Figure 13] FIG. 2 is a diagram showing input images and the like that constitute a cluster. [Figure 14] The feature vectors are shown mapped onto the hypersphere. [Figure 15] FIG. 10 is a diagram showing the correspondence between two-dimensional and three-dimensional representations of feature spaces. [Figure 16] FIG. 10 is a diagram showing detailed learning results (distribution on a hypersphere). [Figure 17] FIG. 10 is a diagram showing a separating plane etc. generated by a linear separator. [Figure 18] FIG. 10 is a diagram showing the concept vectors of each cluster. [Figure 19] FIG. 10 is a diagram showing the concept vectors of each cluster. [Figure 20] FIG. 10 is a diagram showing the concept vectors of each cluster. [Figure 21] FIG. 1 is a conceptual diagram for explaining the similarity between two input images. [Figure 22] FIG. 1 is a diagram showing how a feature vector is projected onto a specific plane (subspace). [Figure 23] FIG. 10 is a diagram illustrating how a feature vector is projected onto a specific line (subspace). [Figure 24] FIG. 10 is a diagram showing how a feature vector is projected onto a specific line (when the concept vector faces the same direction as the x-axis). [Figure 25] 10A to 10C are diagrams showing analysis processing results in the third phase, etc.; [Figure 26] FIG. 10 is a diagram showing a detailed explanation screen for a certain concept. [Figure 27] FIG. 10 is a diagram showing a detailed explanation screen for another concept. [Figure 28] FIG. 10 is a diagram illustrating an outline of a process for generating a concept visualization image. [Figure 29] FIG. 10 is a conceptual diagram showing a process for determining the basis for determining that two images are not similar to each other (basis for determining dissimilarity). DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0020] 1. First Embodiment <1-1. System Overview> Fig. 1 is a schematic diagram showing an image processing system 1. As shown in Fig. 1, the image processing system 1 includes a plurality (a large number) of image capturing devices (such as surveillance cameras) 20 that capture images, and an image processing device 30 that processes the captured images. The image processing device 30 is a device that executes various processes for identifying or classifying the subject (a person in this case) in the captured images. The image processing device 30 can also be expressed as an information processing device that processes various types of information.

[0021] The images captured by each photographing device 20 are input to the image processing device 30 via a communication network (such as a LAN and / or the Internet). Then, the image processing device 30 performs image processing, etc. to identify or classify target persons, etc. in the photographed images. In detail, processing, etc. to identify (recognize) a specific person from among multiple people photographed in multiple photographed images is performed.

[0022] For example, a process is performed to search for images in which a specific person appears (images that include a specific person as a subject) from among multiple images captured by multiple image capture devices 20 arranged in a predetermined area. The multiple image capture devices 20 are distributed and arranged in multiple different locations (different locations along a road, multiple different stores (locations within each store), and / or multiple different locations within the same store (especially a large store)). The image processing device 30 then searches for the specific person to be searched for from the multiple captured images and identifies each image capture device corresponding to one or more of the retrieved images, thereby identifying the behavior (path of travel, etc.) of the specific person within the predetermined area. In short, the image processing device 30 is capable of tracking a specific person. Examples of a specific person include a lost child (e.g., a child) in a lost child tracking process, or a criminal (suspect) in a criminal tracking process. For example, consider a case where the specific person (search target person) is included (photographed) in a total of three images, including an image captured by a certain image capture device 20A, an image captured by another image capture device 20B, and an image captured by yet another image capture device 20C. In this case, the image processing device 30 can determine that the specific person was present in three locations corresponding to the image capture devices 20A, 20B, and 20C. The image processing device 30 can also determine the order in which the specific person moved between the three locations based on the capture time of each image (more specifically, the order in which the images were captured at the three locations).

[0023] Such an inference process, that is, a process of identifying the same person based on multiple images captured by multiple image capture devices 20, is also called person re-identification (re-identification or re-recognition) processing. Note that the tracking process of a specific person related to person re-identification is not limited to criminal tracking processing for tracking a criminal or lost child tracking processing for searching for (tracking) a lost child, but may also be, for example, a tracking process for tracking the behavior of each individual for use in marketing or the like.

[0024] FIG. 2 is a conceptual diagram showing an overview of the processing in the image processing device 30, and FIG. 3 is a flowchart showing the processing in the image processing device 30.

[0025] In this embodiment, as shown in FIGS. 2 and 3, the image processing device 30 first executes a machine learning process (a process of machine learning the learning model 400) for performing the above-described inference process in the first phase PH1 (FIG. 2) (step S11 (FIG. 3)). Specifically, metric learning (also referred to as distance learning) is executed as this machine learning process. More specifically, deep metric learning using a deep neural network (particularly a convolutional neural network) is used. In this metric learning, a learning model 400 is used that outputs a feature vector 250 (251) (see FIG. 7) in a feature space (feature amount space) in response to an input of an input image 210 (211). Such a learning model 400 can also be expressed as a model that indicates a transformation (mapping) from an input image (input) to a feature vector (output). A machine-learned learning model 400 (trained learning model) (420) is generated by the process in the first phase PH1.

[0026] Next, the image processing device 30 executes an inference process as a process of the second phase PH2 (FIG. 2) (step S12). Specifically, the inference process is performed by using the learning model (trained model) 400 (420) learned in the first phase PH1. Specifically, the inference process may include a process of searching for an image including a specific person from among a plurality of captured images 213 (also referred to as gallery images) captured within a predetermined area. More specifically, the inference process may include a process of searching for an image having a similarity to a search source image 215 (also referred to as query image), which is an image of the specific person, at a predetermined level or higher (in other words, the distance between the feature vectors in the feature space is a predetermined distance or less), as an image of the same person as the specific person. Alternatively, the inference process (person re-identification process) may include a process of searching for images similar to the query image 215 in order of similarity. The plurality of gallery images 213 may also be referred to as a group of images constituting a search range (search range image group).

[0027] Furthermore, the image processing device 30 executes a process (such as a process for generating explanatory information) to explain, on a concept basis, the basis for determining the similarity between two images (input images), the query image 215 and the image 213 found in the above-mentioned inference process, as a process of the third phase PH3 (Figure 2) (step S13).

[0028] Specifically, first, prior to deriving the basis for determining the similarity between the two images, the image processing device 30 analyzes what concepts have been acquired (learned) in the machine-learned learning model 400 (also referred to as the trained model 420) (see FIG. 4). Note that FIG. 4 is a flowchart showing such processing (concept analysis processing).

[0029] Next, the image processing device 30 derives the basis for determining the similarity between the two images. In detail, a process is executed to generate information (explanatory information) for explaining the basis for determining the similarity between the two images (the basis for determining that they are similar to each other) on a concept basis (see FIG. 5). Note that FIG. 5 is a flowchart showing such a process (a process for explaining the basis for determining similarity).

[0030] More specifically, among the various concepts acquired by the learning model 400, a concept that has a particularly large influence on the similarity between the two images (a concept with a large contribution) is extracted as a main concept. For example, among multiple concepts, the top several concepts (in order of contribution, etc.) are extracted as main concepts. Then, the concept is determined as the basis for determining the similarity between the two images, and various images representing the concept are displayed on the display unit 35b (presented to the user).

[0031] Such processing (processing in FIGS. 4 and 5, etc.) will be described in detail later.

[0032] <1-2. Image processing device 30> Referring again to Fig. 1, the image processing device 30 includes a controller 31 (also referred to as a control unit), a storage unit 32, a communication unit 34, and an operation unit 35.

[0033] The controller 31 is a control device that is built into the image processing device 30 and controls the operation of the image processing device 30 .

[0034] The controller 31 is configured as a computer system including one or more hardware processors (for example, a central processing unit (CPU) and a graphics processing unit (GPU)). The controller 31 performs various processes by executing, in the CPU or the like, a predetermined software program (hereinafter also simply referred to as a program) stored in a storage unit (a non-volatile storage unit such as a ROM and / or a hard disk) 32. The program (more specifically, a group of program modules) may be recorded on a portable recording medium such as a USB memory, read from the recording medium, and installed in the image processing device 30. Alternatively, the program may be downloaded via a communication network or the like and installed in the image processing device 30.

[0035] Specifically, the controller 31 executes the learning process in the first phase PH1, the inference process in the second phase PH2, and the explanation process (explanation information generation process, etc.) in the third phase PH3.

[0036] The storage unit 32 is configured with a storage device such as a hard disk drive (HDD) and / or a solid state drive (SSD), etc. The storage unit 32 stores the learning model 400 (including learning parameters and programs related to the learning model) (and thus the trained model 420), etc.

[0037] The communication unit 34 is capable of performing network communication via a network. In this network communication, various protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol) are used. By using this network communication, the image processing device 30 can exchange various data (photographed image data, correct answer data, etc.) with a desired destination (for example, the photographing device 20 or an information storage device (not shown)).

[0038] The operation unit 35 includes an operation input unit 35a that accepts operation inputs to the image processing device 30, and a display unit 35b that displays and outputs various information. A mouse, a keyboard, or the like is used as the operation input unit 35a, and a display (such as a liquid crystal display) is used as the display unit 35b. A touch panel that functions as both a part of the operation input unit 35a and a part of the display unit 35b may also be provided.

[0039] The image processing device 30 is also referred to as a learning model generation device because it has the function of machine learning the learning model 400 using training data. The image processing device 30 is also referred to as an inference device because it is a device that performs inference regarding object identification and / or classification using the learning model 400 (420). The image processing device 30 is also referred to as an explanation information generation device because it is a device that generates explanatory information regarding similarity. The image processing device 30 is also referred to as a concept extraction device because it is a device that extracts concepts acquired by the learning model 400 (420), and as a device that explains (analyzes) the basis for the similarity between two images based on the concepts, so it is also referred to as a (similarity) analysis device.

[0040] Furthermore, although various processes (functions) are realized by one image processing device 30 here, this is not limiting. For example, various processes may be shared and realized by multiple devices. For example, the learning process in the first phase PH1, the inference process in the second phase PH2, and the explanation process (explanation information generation process, etc.) in the third phase PH3 described above may each be executed by separate devices.

[0041] <1-3. Learning Phase (First Phase PH1) Processing> As shown in FIG. 3, in this embodiment, a learning process (step S11) in the first phase PH1, an inference process (step S12) in the second phase PH2, and an explanation process (step S13) in the third phase PH3 are executed in this order.

[0042] First, the learning process (step S11) in the first phase PH1 (see FIGS. 2 and 3) will be described below.

[0043] FIG. 7 is a conceptual diagram showing the learning process in the first phase PH1.

[0044] As shown in FIG. 7, in the first phase PH1 (step S11), machine learning processing is performed on the learning model 400 (more specifically, the learning model 410 before learning (FIG. 2)) by metric learning (distance learning). In detail, a plurality of input images 210 (211) in a plurality of teacher data (teacher data group) with correct answer labels are sequentially input to the learning model 400, and an output group (a plurality of feature vectors 250 (251)) from the learning model 400 is obtained (see FIG. 7). Then, a mapping relationship between the input image 210 (input) and the feature vector 250 (output) in the feature space is learned. More specifically, the learning model 400 (mapping relationship) is learned so that the distance in the feature space (the distance between feature vectors) reflects the similarity of the input images in the input space. For example, a learning process is performed to minimize (optimize) an evaluation function such as triplet loss. Through such a learning process, the learning model 400 (410) before learning is learned, and a learned model 420 is generated (step S11).

[0045] More specifically, first, the image processing device 30 generates a person image 210 (also referred to as 211) for machine learning. For example, the image processing device 30 performs a person extraction process and a size adjustment process (resizing process) on each of a plurality of captured images acquired from the image capturing device 20 to generate a plurality of person images 210 (211). The plurality of person images 210 are prepared as a group of input images for the learning model 400. In other words, each person image 210 (211) is prepared as an input image 210 (211) for the learning model 400. For example, a color image (3 channels) having a pixel array (rectangular pixel array) with a width (horizontal) of W0 pixels and a height (vertical) of H0 pixels is prepared as each input image 210. In other words, the input image 210 is generated as voxel data of W0 × H0 × CH0 (where CH0 = 3).

[0046] Furthermore, correct answer information (correct answer label) regarding whether the people in the multiple input images 210 are the same person or different people is assigned to each of the multiple input images 210. For example, a person ID (an identifier for identifying a person) or the like is assigned to each input image 210 (211). In detail, the same person ID is assigned to images of the same person, and different person IDs are assigned to images of different people. In this way, a combination of the correct answer label and the input image 210 (211) is assigned as training data with a correct answer label.

[0047] Next, the multiple input images 210 (input image group) are input sequentially to the learning model 400, and multiple outputs from the learning model 400, i.e., multiple feature vectors 250 (feature vector group) in the feature space, are output sequentially (see Figure 7).

[0048] Here, the learning model 400 has a hierarchical structure in which multiple layers (hierarchies) are hierarchically connected. Specifically, the learning model 400 includes an input layer, multiple intermediate layers, and an output layer. The multiple intermediate layers include a feature extraction layer, etc. The feature extraction layer is configured by repeatedly arranging one or multiple convolution layers and one pooling layer, etc. In each convolution layer, features in the image are extracted by a filter that performs convolution processing. In addition, in each pooling layer, a pooling process (such as average pooling or max pooling) is performed to extract the average pixel value or the maximum pixel value for each small pixel range (for example, a 2 × 2 pixel range), thereby reducing the pixel size (for example, by half in both the vertical and horizontal directions) (condensing the amount of information). A feature map 230 (not shown) is generated by performing multiple feature extraction processes on the input image 210. In addition, by performing a pooling process (e.g., a maximum pooling process) on each channel image of the feature map 230, a feature vector 250 having a predetermined number of channels (number of dimensions) CH1 is generated, and the feature vector 250 is output from the learning model 400.

[0049] As such a learning model 400 (neural network), for example, VGG16 or ResNet (Residual Network) (residual network) may be used. VGG16 is a convolutional neural network model having three convolutional layers, five pooling layers, and three fully connected layers. ResNet (Residual Network) (residual network) is a convolutional neural network that includes summing residuals between layers. The feature extraction layer in ResNet is composed of multiple residual blocks, each of which is composed of a combination of a convolutional layer, an activation function, and skip connections (shortcut connections).

[0050] Various image features of the input image 210 are extracted for each channel in the feature map 230 (in other words, for each channel (element) of the feature vector 250). Note that the image features of the input image 210 are extracted while maintaining their approximate positions within the two-dimensional image of each channel in the feature map 230.

[0051] For example, the feature map 230 is three-dimensional array data (voxel data of W1×H1×CH1) having one channel CH1, each of which is made up of two-dimensional array data of a pixel array (rectangular pixel array) with a width of W1 pixels and a height of H1 pixels. The size (W1×H1) of each channel of the feature map 230 is, for example, 14×14. Each element (value) of the feature vector 250 represents a feature extracted in each channel. The number of dimensions CH1 of the feature vector 250 is the number of channels CH1 of the feature map 230, and is, for example, 1024. However, without being limited to this, the size (W1×H1) of each channel and the number of channels CH1 may be other values. For example, the number of channels CH1 (the number of dimensions of the feature vector 250) may be 512, 2048, or the like.

[0052] Ideally, in the feature space (output space of the learning model 400), multiple feature vectors 250 (251) corresponding to multiple input images 210 (211) of the same person as a subject are located close to each other, and multiple feature vectors 250 corresponding to multiple input images of different people are located far from each other. However, the distribution of the feature vectors based on the output from the learning model 400 before learning (see the rightmost column in Figure 7) deviates from such an ideal distribution state.

[0053] Next, in metric learning, the learning model 400 is trained to optimize (minimize) an evaluation function such as triplet loss. This allows the learning model 400 (mapping relationship) to learn so that the similarity of input images in the input space corresponds to the distance (distance between feature vectors) in the feature space. In other words, the distribution of feature vectors in the feature space gradually changes as the learning progresses. If very good machine learning is performed, the distribution of feature vectors in the feature space gradually approaches the ideal distribution state described above (see the rightmost column in Figure 8). Specifically, in the final feature space, corresponding feature vectors of images of the same person (or people wearing similar clothing) will be distributed relatively close to each other, while corresponding feature vectors of images of different people (or people wearing significantly different clothing) will be distributed relatively far apart. As a result of this machine learning, the learning model 400 before learning (also referred to as 410) changes into the trained learning model 400 (also referred to as 420). The trained model 420 can also be expressed as a feature extractor that extracts features (feature vectors) according to the input image.

[0054] The rightmost columns in each of FIGS. 7 and 8 show how multiple feature vectors 250 (251) are mapped onto a feature space. In the rightmost columns, each feature vector 251 is represented as a single point (more specifically, a point-like figure). More specifically, some of the multiple feature vectors 251 corresponding to the multiple input images 211 are represented as point-like figures (white circles, black circles, hatched white circles, hatched black circles, white squares, black squares, etc.). In each of FIGS. 7 and 8, the rightmost columns (large rectangular portions representing the feature space) and the left portion of the rightmost columns (portions where multiple feature vectors 251 (represented by elongated strip-like rectangles) are arranged) show the same situation. For convenience, multiple points that actually correspond to the same person (and people wearing very similar clothing) are represented by the same figure (point-like figure). However, the image processing device 30 does not know which points (feature vectors (in other words, input images)) actually correspond to the same person (correct labels).

[0055] <1-4. Processing in the inference stage (second phase PH2)> Next, the inference processing in the second phase PH2 (step S12) (see FIGS. 2 and 3) will be described with reference to FIGS. 9 and 10. FIG. 9 is a diagram illustrating the inference processing using the feature vector 250 (253). FIG. 10 is a diagram showing an example of the inference processing result.

[0056] In the second phase PH2 (step S12), the image processing device 30 executes an inference process to identify (or classify) a target (here, a target person) in a plurality of person images in the search range (specifically, a plurality of new input images 210 (213)). Specifically, a person identical to a person in the input image 215 (new input image) of the search target (search source) is searched for among the plurality of new input images 210 (213) captured in the target area (by the image capture device 20 arranged in the target area). In other words, the image processing device 30 identifies (recognizes) a person identical to a person in the input image 215 (query image) of the search target from among the people in the plurality of input images 213.

[0057] For this reason, first, the image processing device 30 inputs a plurality of person images within the search range (specifically, a plurality of new input images 210 (gallery images 213)) into the learning model 420, and obtains an output from the learning model 420. Specifically, as shown in FIG. 9, a feature vector 250 (253) is obtained as an output for each input image 213. Furthermore, each feature vector 250 (253) is generated as, for example, a 1024-dimensional vector. Such a feature vector 253 is obtained for each of the plurality of input images 213 as a vector representing the features of each input image 213 (see the left side of FIG. 9).

[0058] Similarly, the image processing device 30 inputs an input image (query image) 215 to be searched into a learning model 420, and acquires a feature vector 250 (255) output from the learning model 420 (see the right side of FIG. 9). Note that the query image 215 is, for example, an image (such as an image newly assigned for search purposes) different from the multiple input images 213 (gallery images). However, without being limited thereto, the query image 215 may be an image related to a person to be searched that has been discovered (identified) from the multiple input images 213 (gallery images) due to some trigger or the like.

[0059] Next, the image processing device 30 calculates the degree of similarity (for example, Euclidean distance or an inner product between vectors (cosine similarity)) between the feature vector 255 of the query image 215 and each of the plurality of feature vectors 253 related to the plurality of input images 213. The plurality of feature vectors 253 are then sorted in descending order of the degree of similarity (descending order of the degree of similarity). More specifically, the plurality of feature vectors 253 are sorted in ascending order of Euclidean distance (or descending order of cosine similarity).

[0060] For example, the image processing device 30 identifies one or more feature vectors 253 whose distance to the feature vector 255 in the feature space is equal to or less than a predetermined distance (i.e., the degree of similarity is equal to or greater than a predetermined level) as the feature vector 255 of the same person as the person in the query image 215. In other words, the image processing device 30 recognizes the person in one or more input images 213 corresponding to the identified one or more feature vectors 255 as the same person as the person in the query image 215.

[0061] FIG. 10 shows how a plurality of feature vectors 253 (represented by open circles with sand-hatched areas in FIG. 10 ) corresponding to a plurality of input images 213 are distributed in a feature space. In FIG. 10 , three feature vectors 253 (V301, V302, V303) exist within a predetermined distance range from a feature vector 255 (see open stars) of a query image 215. In this case, for example, the three images 213 corresponding to the three feature vectors 253 (V301, V302, V303) are extracted as images of the same person. The three feature vectors 253 are also arranged in descending order of similarity to the feature vector 255 (ascending order of distance). Here, the three person images 213 corresponding to the top three feature vectors 253 are recognized as images of the same person (or a person very similar) as the person in the query image 215.

[0062] However, without being limited to this, a person in the input image 213 corresponding to a top predetermined number of feature vectors 250 (253) sorted in ascending order of the distance may be recognized as the same person as the person in the query image 215. Alternatively, the multiple input images 213 may simply be sorted in ascending order of distance (related to the feature vector 255) from the query image 215 (descending order of similarity). Even in this case, the image processing device 30 essentially executes a process of searching for a person who is highly likely to be the same person as the person in the query image in order of likelihood (a process of recognizing the same person), and this process is one type of inference process for recognizing a target person in the query image.

[0063] Here, each of the plurality of feature vectors 250 (251, 253, 255) is assumed to be normalized (specifically, L2 normalized). As an index indicating the similarity between any two of the plurality of feature vectors 250, the inner product (in other words, cosine similarity) between two feature vectors 250 (also denoted as F) is adopted. Specifically, the inner product (in other words, cosine similarity) between two feature vectors F (F) corresponding to two input images X (query image Xq and gallery image Xg) is adopted. q ,F g ) dot product (=F q F g =q·g) is used as the similarity measure between the two input images. q is the input image (query image) X q is the output vector (feature vector in the feature space (output space)) F from the learning model 400 for g is an input image (a gallery image) X g The output vector F from the learning model 400 for the feature vector F is shown in Fig. 1. The symbol "·" represents an inner product. q is also simply expressed as the feature vector q, and the feature vector F g is also simply expressed as a feature vector g. Such similarity St is expressed as in the following equation (1).

[0064]

number

[0065] Note that because each feature vector F(250) is normalized (the magnitude of each vector F is 1), the dot product between two feature vectors is equal to the cosine similarity between the two feature vectors. Furthermore, a large cosine similarity (and dot product) (close to 1) means that the angle θ between the two feature vectors F is small (the two feature vectors F are similar), and therefore the two input images corresponding to the two feature vectors F are similar. In other words, the greater the similarity St between the two feature vectors F (the dot product of the two feature vectors F), the more similar the two input images corresponding to the two feature vectors F are.

[0066] <1-5. Distribution of feature vector F> Here, in FIGS. 7 and 8 (the rightmost columns), the distribution of feature vectors F is expressed two-dimensionally as a group of points on a plane (hyperplane).

[0067] On the other hand, as shown in Fig. 14 etc., the distribution of feature vectors F can also be expressed three-dimensionally as a group of points on a sphere (hypersphere). In the following, the latter expression (three-dimensional expression using a hypersphere) will be mainly used for explanation.

[0068] Fig. 14 shows a state in which multiple (here, three) feature vectors F(250) output from the trained model 420 are mapped onto a hypersphere. Here, each feature vector F(250) is normalized (the norm (magnitude) of each vector F is 1). Therefore, as shown in Fig. 14, the feature vector F can be expressed as a vector starting from the origin and ending at a point on the hypersphere.

[0069] Furthermore, as mentioned above, a small angle θ between two feature vectors F (a large inner product of the two feature vectors F) means that the two input images corresponding to the two feature vectors F are similar.

[0070] Fig. 15 is a diagram showing the correspondence between two-dimensional and three-dimensional representations of feature space. In Fig. 15, of the four groups of figures roughly divided into left, right, top, and bottom, the two groups of figures at the top of Fig. 15 are diagrams showing how each feature vector F(250) is distributed on a hyperplane. In contrast, the two groups of figures at the bottom of Fig. 15 are diagrams showing how each feature vector F(250) is distributed on a hypersphere.

[0071] In detail, the group of figures (spheres and point figures on their surfaces, etc.) at the bottom left of Fig. 15 show the distribution of output vectors (feature vectors F) by the "untrained learning model 400" in the feature space, similar to the rightmost column of Fig. 7. The group of figures at the bottom left illustrates the feature space three-dimensionally, and shows a situation similar to that of the group of figures (large rectangles and figures inside them) at the top left of Fig. 15 (however, three-dimensionally, not two-dimensionally).

[0072] 15 (spheres and point-like figures on their surfaces, etc.), similar to the rightmost column of FIG. 8, show the distribution of output vectors (feature vectors F) from the "trained learning model 400 (420)" in the feature space. The figures in the lower right illustrate the feature space three-dimensionally, and similar to the figures in the upper right corner of FIG. 15 (large rectangles and figures inside them), This shows the situation (but in three dimensions, not two dimensions).

[0073] For convenience, the upper part of Fig. 15 and Fig. 8 are conceptual diagrams that represent the distribution of feature vectors F in two dimensions, while the lower part of Fig. 15 and Fig. 14 are conceptual diagrams that represent the distribution of feature vectors F in three dimensions. An actual feature vector F is usually not a two- or three-dimensional vector, but a very high-dimensional (even higher) multidimensional vector (for example, a 1024-dimensional vector). It is difficult to visually illustrate a vector with four or more dimensions in three-dimensional space, and these graphical representations of feature vectors F (see Fig. 8, Fig. 14, Fig. 15, etc.) are all simplified virtual representations.

[0074] <1-6. Details of the first and second phases of processing> In this embodiment (particularly the above-described first phase PH1 and second phase PH2), more specifically, the following person re-identification (re-identification) processing is executed.

[0075] First, in the first phase PH1 (step S11), the learning model 400 is machine-trained based on a large number of images (e.g., thousands to hundreds of thousands) each containing a large number of people (e.g., hundreds to tens of thousands). Note that it is assumed that the people in the large number of images are wearing different clothes, and that no two people are wearing exactly the same clothes. Learning is performed so that feature vectors corresponding to images of people wearing similar clothes are located relatively close to each other (more closely than feature vectors corresponding to images of people wearing different clothes) in the feature space (the output space of the learning model 400).

[0076] Specifically, the system is trained so that feature vectors corresponding to images of the same person (the same person wearing the same clothes) are positioned very close to each other in the feature space. In other words, the system is trained so that multiple images of the same person with certain clothing characteristics are positioned very close to each other in the feature space. That is, images of the same person (specifically, their feature vectors) are positioned very close to each other in the feature space. Furthermore, images of different people wearing similar clothing (different people with similar clothing characteristics) (specifically, their feature vectors) are also positioned close to each other in the feature space. On the other hand, images of different people wearing very different clothing (specifically, their feature vectors) are positioned relatively far apart in the feature space.

[0077] An example of such a learning result is shown in the rightmost column of Figure 8. In detail, each output vector (feature vector F) from the trained model 420 is represented by a dotted figure, and multiple dotted figures representing multiple feature vectors F are plotted (arranged) in (a large rectangle representing) the feature space.

[0078] 11 and 16 are diagrams showing an example of a more detailed learning result (than the rightmost column of FIG. 8). In the large rectangle extending from the center to the lower half of FIG. 11, the distribution of output vectors (feature vectors F) by the trained learning model 400 (trained model 420) is shown two-dimensionally. FIG. 16 is a diagram showing a similar situation three-dimensionally.

[0079] As described above, in the feature space, multiple (e.g., two) feature vectors F corresponding to multiple (e.g., two) images of the same person are arranged very close (relatively close). For example, in FIGS. 11 and 16, two point-like figures (e.g., a pair of cross-hatched white circle figures) are arranged very close to each other. This indicates that two feature vectors F corresponding to two images of the same person (both images of the person wearing the same clothes) are arranged very close to each other. In other words, the pair of point-like figures corresponds to a pair of images of the same person wearing the same clothes.

[0080] Furthermore, images of different people wearing similar clothing (specifically, their feature vectors F) are also arranged close to each other in the feature space (although not as close as images of the same person). For example, in Figures 11 and 16, nine pairs of white circles (a total of 18 white circles) are arranged relatively close to each other. Of the nine pairs of white circles (both with and without hatching), six pairs of white circles (with hatching) are particularly arranged relatively close to each other.

[0081] On the other hand, images of people who are significantly different from each other (specifically, their feature vectors F) are located relatively far apart in the feature space. For example, a pair of black circles and a pair of white circles (without hatching) are located far apart.

[0082] In this way, the feature vectors F for the same person are clustered in a relatively small area, while the feature vectors F for different people wearing similar clothing are clustered in a slightly wider area. On the other hand, the feature vectors F for very different people (different people wearing very different clothing) are clustered relatively far apart (with a relatively large dispersion).

[0083] For convenience of illustration, each figure shows only some of the feature vectors F among the multiple feature vectors F. In particular, in Figures 8, 15, 17 and subsequent figures, even fewer feature vectors F are shown than in Figures 11 and 16, etc.

[0084] Next, in the second phase PH2 (step S12), the image processing device 30 uses the trained learning model 400 (420) to determine whether two people (the person in the query image 215 and the person in the gallery image 213) are the same person (the degree of similarity between the two people). It is assumed that the same person wears the same clothes, and people wearing similar clothes are searched for as the same person (more specifically, people who are likely to be the same person). The similarity between pairs of images of people wearing similar clothes is relatively high.

[0085] Specifically, for example, a combination of feature vectors F (and consequently a corresponding image pair) that maximizes the similarity St shown in equation (1) is found. In particular, a gallery image corresponding to the feature vector F that maximizes the similarity St with the feature vector F of the query image is extracted. That is, two images whose feature vectors are similar to each other are extracted as images of the same person. For example, as shown in FIG. 21 (or FIG. 14), people wearing the same clothes (or similar clothes) are presumed to be the same person and extracted.

[0086] Furthermore, in the next third phase PH3 (step S13), the image processing device 30 executes a process to explain the basis for the image processing device 30's determination that the two people are the same person (or similar people). For example, it is analyzed whether the basis for determining the similarity is that the two people are wearing white shorts and / or that they are wearing a patterned (checked, etc.) shirt. Furthermore, as a preliminary step, the trained model 420 executes an analysis process to determine what concepts it has learned based on the training data. These will be described next.

[0087] <1-7. Explanation Stage (Phase 3 PH3) Processing (Overview)> In this embodiment, the image processing device 30 further executes the processing of the third phase PH3 (FIG. 2). The processing of the third phase PH3 is a processing (processing for generating explanatory information, etc.) for explaining the basis for determining the similarity between two images (input images) on a concept basis. Here, the case where the similarity between the query image 215 and the image 213 found in the above inference processing is determined as the similarity between two images will be mainly described.

[0088] The processing of this third phase PH3 is roughly divided into processing of sub-phase PH3a (step S20 (FIG. 4)) and processing of sub-phase PH3b (step S30 (FIG. 5)).

[0089] In the former sub-phase PH3a (step S20 (FIG. 4)), an analysis process is performed to analyze what concepts have been acquired (learned) in the machine-learned learning model 420. The former sub-phase PH3a is a preprocessing for the latter sub-phase PH3b.

[0090] Specifically, in sub-phase PH3a, multiple feature vectors F are acquired in a feature space output from the learning model 400 in response to multiple input images (e.g., multiple input images 211 used in machine learning) input to the learning model 400. Then, a hierarchical clustering process is performed on the multiple feature vectors F to generate multiple hierarchical clusters G. Furthermore, a vector corresponding to a specific cluster among the multiple clusters (a representative vector for the specific cluster) is extracted as a concept of the specific cluster (more specifically, a vector representing the concept, etc.). For example, a concept activation vector CAV (described later) for the specific cluster is used as the representative vector for the specific cluster. As described later, a subspace corresponding to the specific cluster (e.g., a subspace spanned by the representative vector) may be extracted as the concept of the specific cluster.

[0091] Meanwhile, in the latter sub-phase PH3b (step S30 (FIG. 5)), processing is performed to derive a basis for determining the similarity between two images. Specifically, the influence of various concepts (concepts obtained in sub-phase PH3a (FIG. 4)) on the similarity between the two images is evaluated. More specifically, the contribution of each of the various concepts (multiple concepts extracted from the learning model 400) acquired by machine learning by the learning model 400 to the similarity between the two images is calculated. Then, based on the contribution, etc., a main concept among these concepts is identified. Furthermore, a screen display explaining the main concept is performed.

[0092] Below, the sub-phases PH3a and PH3b will be explained in this order.

[0093] <1-8. Processing of Subphase PH3a (Step S20)> First, the processing of sub-phase PH3a will be described with reference to FIG.

[0094] <Step S21> As shown in FIG. 4, first, in step S21, the controller 31 (image processing device 30) acquires a plurality of feature vectors 251 (in feature space) output from the learning model 420 in response to a plurality of input images 210 (211) input to the machine-learned learning model 420. Here, the training data used in the machine learning of the trained model 420 (more specifically, the input images 211 constituting the training data) is used as the input images 210 to the trained model 420. The plurality of feature vectors 251 can also be expressed as feature vectors that are finally output at the final stage of the learning process for the learning model 400 (after the learning process involves many iterations).

[0095] Each feature vector 251 is a vector (a vector with a dimension CH1, for example, a 1024-dimensional vector) output from the trained model 420 in response to input of each input image 211 to the trained model 420. As described above, the multiple feature vectors 251 obtained in this manner are distributed at appropriate positions in the feature space by the trained trained model 420. As a result of metric learning for the trained model 400, the distances between the multiple feature vectors 251 in the feature space reflect the similarity of the corresponding input images in the input space (see the lower part of Figure 11 and Figure 16, etc.). The distribution of the multiple feature vectors 251 in the feature space can also be considered to be a distribution based on concepts acquired by training. Specifically, feature vectors corresponding to the same concept and feature vectors corresponding to similar concepts are considered to be distributed relatively close to each other in the feature space.

[0096] <Step S22: Hierarchical clustering process> Next, in step S22, the controller 31 generates a plurality of hierarchical clusters by executing a hierarchical clustering process on the plurality of feature vectors 251 based on the positional relationships in the feature space, etc. The hierarchical clustering process is also called a hierarchical clustering process or a hierarchical clustering process.

[0097] The hierarchical clustering process is a process of sequentially grouping a plurality of elements (elements of a set) (here, the feature vector F(251)) to form hierarchical clusters (groups).

[0098] Specifically, in the hierarchical clustering process, the most similar (highest mutual similarity) tentative clusters (described below) are sequentially combined one by one to generate clusters (until the whole is reduced to one cluster). In this way, hierarchical clusters are formed. A tentative cluster (provisional cluster) is initially composed of a single feature vector F, and thereafter is composed of a single feature vector F or two or more feature vectors F. A tentative cluster composed of two or more feature vectors F means a (new) cluster generated in the hierarchical clustering process.

[0099] In the hierarchical clustering process, whether two provisional clusters are similar to each other is determined based on the distance between the two provisional clusters (for example, Euclidean distance or cosine similarity). Various methods, such as the centroid method, the shortest distance method, the longest distance method, the group average method, or Ward's method, may be used for the hierarchical clustering process. These methods are classified according to the specific quantity used to calculate the distance (similarity) between the two provisional clusters. For example, the centroid method is a method in which the distance between the centroids of the two provisional clusters is used as the distance (similarity) between the two provisional clusters. Furthermore, the shortest distance method is a method in which the shortest inter-element distance between any element in one provisional cluster and any element in the other provisional cluster is used as the distance between the two provisional clusters. In each method, the index value representing the similarity is not limited to distance, and cosine similarity or the like may also be used. The following mainly describes the case in which cosine similarity (see Equation (1)) is used.

[0100] More specifically, in the hierarchical clustering process, first, evaluation values ​​(specifically, similarity St (see formula (1)) etc.) are calculated for combinations of all provisional clusters (initially, individual feature vectors F). Then, a new cluster is formed by combining the provisional clusters (element pairs) with the highest evaluation value (similarity St). Then, by repeating the same process, new clusters (same-level clusters or higher-level clusters) are formed one after another, and finally, one large cluster is formed. The inclusion relationships (in other words, hierarchical relationships) of the formed clusters are represented by a tree diagram (also called a dendroid) (see the top part of Figure 11).

[0101] The lower part of Figure 11 (and Figure 16), similar to Figure 8, shows how multiple feature vectors F (251) are distributed in a feature space (the output space of the trained model 420). The multiple feature vectors 251 having such a distribution form multiple clusters hierarchically organized in multiple layers. The upper part of Figure 11 also shows a tree diagram (dendroid) for the multiple feature vectors F.

[0102] For example, in the middle stage of the hierarchical clustering process described above, a cluster G310 (see the center left side of FIG. 11) is formed. Cluster G310 is composed of multiple (six in FIG. 11) feature vectors 251 (dotted black circle shapes) that are close to each other. Furthermore, in a subsequent stage, a cluster G300 is formed that encompasses cluster G310. Cluster G300 is composed of feature vectors 251 in cluster G310 and other feature vectors 251 (hatched dotted black circle shapes) that are close to cluster G310. Cluster G300 is also expressed as a higher-level cluster (also referred to as a parent cluster) of cluster G310.

[0103] Similarly, at a certain intermediate stage of the hierarchical clustering process, clusters G111, G112, and G120 are formed (see the lower center of FIG. 11). Cluster G111 is composed of multiple (six in FIG. 11) feature vectors 251 (dotted white circles with vertical and horizontal cross (checkerboard) hatching) that are close to each other. Cluster G112 is composed of multiple (six in FIG. 11) feature vectors 251 (dotted white circles with diagonal cross hatching) that are close to each other. Cluster G120 is composed of multiple (six in FIG. 11) feature vectors 251 (dotted white circles without hatching) that are close to each other. At a subsequent stage, cluster G110 (a higher-level cluster of the two clusters G111 and G112) is formed, which includes both clusters G111 and G112. At a later stage, a cluster G100 (a higher cluster than the two clusters G110 and G120) is constructed that includes both clusters G110 and G120. At an even later stage, a cluster G10 (a higher cluster than the two clusters G100 and G200) is constructed that includes both clusters G100 and G200.

[0104] Furthermore, other clusters G400, G510, G500, etc. are also formed as the hierarchical clustering process progresses.

[0105] <Results of hierarchical clustering processing> By such hierarchical clustering processing, a plurality of clusters (G111, G112, G110, G120, G100, G200, G10, G310, G300, G400, G510, G500, etc.) are formed, for example, as shown in Fig. 11. Note that, for convenience of illustration, only some of the many feature vectors 251 are shown in Fig. 11, and only some of the many clusters are shown.

[0106] Furthermore, each feature vector F (251) corresponds to a respective input image X (211). Therefore, the hierarchical clustering process is a process of clustering a plurality of feature vectors F (251) and also a process of clustering a plurality of input images X (211) (see FIG. 13). FIG. 13 is a diagram showing some of the clusters shown in FIG. 11. FIG. 13 shows input images 210 (211) corresponding to the some of the clusters. Note that although the input image 210 is actually a photographed image, for convenience of illustration, the input image 210 (211, 213, 215) is represented as a CG (computer graphics) image in FIG. 13 (as well as the subsequent figures (FIGS. 14, 21, 25, 26, etc.)).

[0107] For example, cluster G111 can be expressed as including a plurality of input images corresponding to a plurality of dotted white circle figures with vertical and horizontal cross (checkerboard) hatching (see FIG. 11) (see also FIG. 13). Similarly, cluster G112 includes a plurality of input images corresponding to a plurality of dotted white circle figures with diagonal cross hatching (see FIG. 11). Furthermore, upper cluster G110 includes both a plurality of input images included in lower cluster G111 and a plurality of input images included in lower cluster G112.

[0108] More specifically, as shown in FIG. 13, cluster G111 is made up of a plurality of images 211 of at least one person (three or more in this case) wearing a "heavily patterned white (white background) shirt (where the pattern is linear)." Cluster G112 is made up of a plurality of images 211 of at least one person wearing a "heavily patterned white shirt (where the pattern is curved)." Upper cluster G110 is made up of both person images 211 included in lower cluster G111 and person images 211 included in lower cluster G112. More specifically, upper cluster G110 is made up of a plurality of images 211 of at least one person wearing a "heavily patterned white shirt" (where the pattern may be linear or curved).

[0109] Furthermore, cluster G120, which has a peer relationship with cluster G110 (the cluster corresponding to the image of a person wearing a "white shirt with a lot of pattern"), is composed of multiple images 211 of at least one person wearing a "white shirt with a little pattern."

[0110] Furthermore, the upper cluster G100 is configured with both the person images 211 included in the lower cluster G110 and the person images 211 included in the lower cluster G120. More specifically, the upper cluster G100 is configured with a plurality of images 211 of at least one person wearing a "white shirt with a pattern" (the pattern may be heavy or light).

[0111] Furthermore, cluster G200, which has a peer relationship with cluster G100 (the cluster corresponding to the image of a person wearing a "patterned white shirt"), is composed of multiple images 211 of at least one person wearing a "light pink shirt."

[0112] Other clusters are similarly composed of images of people dressed similarly to each other for a particular concept.

[0113] In this way, clustering the plurality of feature vectors 250 (251) corresponding to each input image 210 (211) is equivalent to clustering the plurality of input images 210.

[0114] In other words, a plurality of input images belonging to a specific cluster (generated by the hierarchical clustering process) have common features, and the specific cluster is interpreted as having a unique concept.

[0115] Furthermore, the concepts of a higher-level cluster are concepts (overarching concepts) that encompass (subsume) the concepts of its lower-level clusters. Conversely, the concepts of lower-level clusters are concepts that subdivide the concepts of their higher-level clusters. In short, higher-level concepts are coarse-grained concepts, while lower-level concepts are fine-grained concepts. It can also be said that higher-level concepts are concepts shared by a relatively large number of people, while lower-level concepts are concepts shared by a relatively small number of people.

[0116] Note that the tree diagram (dendroid) in Figure 11 is just an example, and a different tree diagram will be generated depending on the training data, etc.

[0117] Furthermore, such a hierarchical clustering process generates a very large number of clusters. All clusters generated by the hierarchical clustering process may be used in subsequent processes (particularly, processes after step S31 (described later)), but this is not limited to this. For example, among such a large number of clusters, it is preferable to use clusters containing a predetermined number of people (e.g., three people) or more as clusters that indicate the explanatory basis of similarity (clusters that form the concept of the explanatory basis). This makes it possible to improve the robustness of the concept (by suppressing dependence on characteristics unique to people in the training data). In other words, clusters consisting of only a single person may be excluded from clusters that indicate the explanatory basis of similarity.

[0118] In particular, it is preferable to use only a plurality of clusters whose concept vectors U (described later) are linearly independent from one another among all the clusters generated by the hierarchical clustering process.

[0119] <Step S23: Extraction of Concept Vector U> In the next step S23, the controller 31 extracts a subspace or vector corresponding to each cluster as a concept of the cluster (step S23). Here, a vector ("concept vector" (described next)) corresponding to each cluster is extracted as a concept of the cluster.

[0120] More specifically, the controller 31 first selects, from all clusters, clusters that can be considered in subsequent processing. Specifically, from all clusters generated by the hierarchical clustering process, some clusters that satisfy a predetermined criterion are selected. The selected clusters are also referred to as candidate clusters to be considered. The predetermined criterion may be, for example, that the clusters contain a predetermined number of people (e.g., three people) or more (as described above). Alternatively, all clusters may be selected as candidate clusters to be considered.

[0121] Then, the controller 31 extracts each vector U corresponding to each of the selected clusters (plurality of specific clusters) as a concept of the cluster (each specific cluster) (step S23). Specifically, the vector U corresponding to a specific cluster is a representative vector for the specific cluster. Note that the "representative vector" does not need to be any of the plurality of feature vectors belonging to the specific cluster, but may be a vector that representatively indicates a group made up of the plurality of feature vectors (a representative vector that symbolizes the plurality of feature vectors belonging to the specific cluster). Furthermore, since the vector U is a vector that expresses the concept of the specific cluster (a vector expression of the concept), it is also referred to as a "concept vector." Like the feature vector F, the concept vector U is preferably normalized.

[0122] The concept vector U of a specific cluster is calculated as the average vector of the multiple feature vectors F(251) that belong to (constitute) the specific cluster. The average vector is also called the centroid vector because it indicates the centroid position (average position) of the multiple feature vectors F on the hyperplane (hypersphere) of the feature space.

[0123] Alternatively, the concept vector U of a specific cluster may be the "concept activation vector" (CAV) of the specific cluster. The concept activation vector of a specific cluster is the normal vector of a separating plane 501 that separates elements belonging to the specific cluster from other elements (see FIG. 17). FIG. 17 shows the separating plane 501 in a feature space (here, a hypersphere). The separating plane 501 is a plane that separates elements belonging to the specific cluster Ga (see the dotted white circle (feature vector Fa)) from elements not belonging to the specific cluster Ga (see the other dotted circle (feature vector Fb)). Such a separating plane 501 can be obtained by a two-class linear separator (linear classifier) ​​(such as a support vector machine). A vector perpendicular to (and pointing outward from) this separating plane 501 is the concept activation vector (CAV) of the specific cluster Ga.

[0124] 18 to 20 are diagrams showing the concept vectors U (CAV, etc.) of each cluster. In each diagram, a certain cluster G is shown on the left, and the concept vector U of the certain cluster G (concept vector expressed as a vector pointing to a point on a hypersphere) is shown on the right.

[0125] For example, the concept vector U(U110) of cluster G110 is shown in the upper part of Fig. 18, and the concept vector U(U120) of cluster G120 is shown in the middle part of Fig. 18. Furthermore, the concept vector U(U100) of cluster G100 is shown in the lower part of Fig. 18.

[0126] Moreover, the concept vector U (U200) of cluster G200 is shown in the upper part of Fig. 19, and the concept vector U (U310) of cluster G310 is shown in the middle part of Fig. 19. Moreover, the concept vector U (U300) of cluster G300 is shown in the lower part of Fig. 19.

[0127] Similarly, the concept vector U(U400) of cluster G400 is shown in the upper part of Fig. 20, and the concept vector U(U510) of cluster G510 is shown in the middle part of Fig. 20. Furthermore, the concept vector U(U500) of cluster G500 is shown in the lower part of Fig. 20.

[0128] Although not shown here, other concept vectors U, for example, concept vectors U111, U112, U10, etc., are also calculated in the same manner.

[0129] As shown in these figures, the concept vectors U of clusters that are close to each other in the dendroid (see the top row of FIG. 11) are relatively similar. For example, the concept vector U110 of cluster G110 (see the top row of FIG. 18) and the concept vector U120 of cluster G120 (see the middle row of FIG. 18) are relatively close to each other (have relatively similar orientations). Conversely, the concept vectors U of clusters that are far apart from each other in the dendroid (see the top row of FIG. 11) are significantly different from each other. For example, the concept vector U100 of cluster G100 (see the bottom row of FIG. 18) and the concept vector U500 of cluster G500 (see the bottom row of FIG. 20) are significantly different from each other (have significantly different orientations).

[0130] In this way, the similarity of the concept vector U reflects the similarity of the features of each cluster (in other words, the concepts of each cluster).

[0131] Furthermore, as described above, multiple input images belonging to a specific cluster (generated by the hierarchical clustering process) have common features, and the specific cluster is interpreted as having a unique concept.

[0132] Therefore, the controller 31 extracts the concept vector U of each cluster as the concept (or concept expression) of that cluster. The concept vector U of each cluster is a vector that expresses the concept of that cluster. In other words, the concept vector U of each cluster is also a "vector-based concept expression" of that cluster.

[0133] As described above, the trained model 420 is interpreted as a model that has learned the concepts of each cluster (generated by the hierarchical clustering process). Then, the concept vector U of each cluster is extracted as the concept of that cluster.

[0134] As described above, each concept vector U is expressed three-dimensionally in Fig. 16 and Figs. 18 to 20, etc. However, in reality, each concept vector U is a vector (multidimensional vector) of a very high dimension (β dimension (e.g., 1024 dimension)). Therefore, a very large number (γmax) (e.g., 300) (where γmax<β)) of concept vectors U corresponding to different clusters (concepts) exist as linearly independent vectors.

[0135] Furthermore, there is a limit to the visualization (three-dimensional representation) of such multidimensional (e.g., 1024-dimensional) concept vectors U. Each concept vector U often has substantial feature components in dimensions other than the three dimensions that can be visualized (e.g., the i-th dimension from the fourth dimension onward). For example, in Fig. 16, Fig. 18 to Fig. 20, etc., multiple concept vectors U that are significantly different from one another have different orientations within the three dimensions. However, in reality, multiple concept vectors U that are significantly different from one another often have different orientations in the i-th dimension from the fourth dimension onward.

[0136] <1-9. Similarity of two images (similarity Sc for each concept group, etc.)> Before describing the processing of sub-phase PH3b (FIG. 5), the similarity between two images (an image pair) will be described. The two images (first and second images) are exemplified by a query image and a gallery image extracted in step S12. However, the present invention is not limited to this, and the similarity between any two images can be determined in the same manner.

[0137] First, as described above, the similarity St between two input images (overall similarity St between an image pair) is expressed by the above formula (1). That is, the similarity St between two input images X is calculated (expressed) as the inner product (q·g) (here, cosθ) of the output vectors q and g (feature vectors F) from the learning model 400 for the two input images X. For example, the similarity St between two input images (query image and gallery image) shown in FIG. 21 is calculated as the inner product of the respective feature vectors q and g (St=0.778 in FIG. 21).

[0138] Here, we introduce another index (specifically, similarity Sc) different from the similarity St (overall similarity). This similarity Sc is an index value indicating the degree of similarity (between two images) explained by a concept (also referred to as a concept under consideration or a selected concept) selected from among multiple concepts as a target for considering the degree of contribution to the similarity. In short, the similarity Sc does not represent the overall similarity between two input images, but rather represents the partial similarity resulting from some selected concepts. In other words, the similarity Sc represents the component contributed by some selected concepts (concepts under consideration) of the overall similarity St between two input images. In other words, the similarity Sc is an index indicating the degree of contribution (contribution) of the selected concept to the similarity between both images. The similarity Sc can also be expressed as the similarity (contribution) for each concept (for each concept vector or for each cluster). Note that the similarity Sc, like the similarity St, is a scalar (value).

[0139] This similarity Sc for the selected concept is expressed as the degree of similarity between image features (feature vectors Fq (= q) and Fg (= g)) in a subspace (also referred to as a specific subspace) spanned by the concept vector U corresponding to the selected concept. Specifically, the similarity Sc is expressed as the inner product of a vector (Pq) obtained by projecting (orthogonally projecting) a first feature vector q onto the specific subspace, and a vector (Pg) obtained by projecting (orthogonally projecting) a second feature vector g onto the specific subspace. The vector (Pq) is a vector (orthogonal projection vector) obtained by applying a projection matrix P (described below) to the feature vector q (from the left), and the vector Pg is a vector (orthogonal projection vector) obtained by applying the projection matrix P to the feature vector g (from the left).

[0140] That is, the similarity Sc is expressed by the following formula (2).

[0141]

number

[0142] Here, the matrix P is a specific projection matrix (specifically, an orthogonal projection matrix) that projects each feature vector F onto the specific subspace (the subspace spanned by the concept vector U corresponding to the selected concept). Specifically, the projection matrix (orthogonal projection matrix) P is calculated using the matrix B according to the following equation (3). The matrix P has a size of β×β. The value β is the number of dimensions (e.g., 1024) of the vector F (q or g).

[0143]

number

[0144] Here, matrix B is a matrix in which a predetermined number (the number of concepts to be selected (γ)) of concept vectors U (vertical vectors (column vectors)) are arranged horizontally. Matrix B has a size of β×γ (for example, 1024×2). Furthermore, γ is the number of concepts to be selected (for example, 2 (or 1 or 3, etc.)). Furthermore, the subscript "T" in the upper right corner of the matrix indicates that it is a transpose (matrix).

[0145] The γ concept vectors U are selected so as to be linearly independent of each other. The maximum value of the number γ of selected concepts is β.

[0146] Equation (3) is also a general equation for finding a matrix (orthogonal projection matrix) that represents an orthogonal projection onto a subspace spanned by one or more vectors b (a subspace based on two or more vectors b). However, the concept vector U is used as vector b.

[0147] Matrix B (and hence matrix P) will differ depending on the selected concepts (specifically, the concept vectors corresponding to the selected concepts). Specifically, matrix B (and matrix P) will vary depending on which concepts (and how many concepts) are selected.

[0148] For convenience in calculating the inverse matrix, the projection matrix P may be calculated using the following equation (4) instead of equation (3). The term +εE is an adjustment term to prevent divergence to infinity, etc. The matrix E is a unit matrix of dimension β (unit matrix of β × β (size)), and the value ε is a tiny constant.

[0149]

number

[0150] Referring again to equation (2), equation (2) will be explained.

[0151] In equation (2), as described above, two feature vectors F (specifically, vectors q and g) corresponding to two input images X (Xq, Xg) are transformed into two projection vectors (PF) using a specific projection matrix P. The projection vectors (PF) are vectors obtained by applying the projection matrix P to the feature vectors F (from the left). As described above, the specific projection matrix P is a projection matrix (orthogonal projection matrix) that projects (orthogonally projects) each feature vector in the feature space onto a specific subspace (a subspace corresponding to a specific concept).

[0152] This specific projection matrix P (see equation (3) etc.) is defined for each concept group consisting of n selected concepts (one or more concepts) to be evaluated.

[0153] For example, when the concept to be evaluated is a single concept, the specific projection matrix P is a projection matrix that projects each feature vector F in the feature space onto a specific line (specific subspace). More specifically, the specific projection matrix P for a single concept defined by a single concept vector U is a projection matrix that projects each feature vector in the feature space onto a specific line (a line that includes the single concept vector U).

[0154] Figure 23 shows how each feature vector q, g is projected (orthogonally projected) onto a specific line (a line containing a single concept vector U1) using such a projection matrix P (which is an orthogonal projection matrix onto a subspace (i.e., a line) spanned by a single concept vector U1).

[0155] As shown in Fig. 23, vector q is transformed into vector q1 and vector g is transformed into vector g1 by this orthogonal projection matrix P. In Fig. 23, the transformed vectors q1 and g1 are indicated by thick dashed lines. In this way, vectors q1 and g1 are projection vectors obtained by projecting two feature vectors q and g onto a subspace (specifically, a straight line) corresponding to a specific cluster (specific concept vector).

[0156] Therefore, equation (2) is transformed into the following equation (5): Specifically, the inner product between the two projection vectors q1 and g1 (q1·g1=qs*gs) is calculated as the similarity Sc for each cluster (concept).

[0157]

number

[0158] Here, the value qs is the magnitude of vector q1 (the projected component of vector q projected onto a line in the direction of concept vector U), and the value gs is the magnitude of vector g1 (the projected component of vector g projected onto a line in the direction of concept vector U). However, if the angle between vector q1(, g1) and concept vector U is between 90 degrees and 270 degrees, the value qs(, gs) is a negative value. The similarity Sc shown in equation (2) is calculated as the product of the value qs and the value gs, as shown in equation (5).

[0159] FIG. 24 is similar to FIG. 23. However, FIG. 24 illustrates a case where the concept vector U is oriented in the same direction as the x-axis. In this case, as shown in FIG. 24, the value qs is equal to the x-direction component (first component) of the vector q1, and the value gs is equal to the x-direction component (first component) of the vector g1. Although FIG. 24 is shown for the sake of simplicity, a projective transformation such as that shown in FIG. 23 is generally assumed.

[0160] According to formula (2) (particularly formula (5)), the contribution of a single concept vector U (in other words, the contribution of a single selected concept to the similarity between both images) is calculated as the similarity Sc. In other words, the similarity Sc for each concept is calculated.

[0161] Furthermore, when there are two concepts to be evaluated, the specific projection matrix P is a projection matrix that projects each feature vector F in the feature space onto a specific plane (a specific subspace). More specifically, the specific projection matrix P for the two concepts is a projection matrix that projects each feature vector F in the feature space onto a specific plane (a plane spanned by two concept vectors U1 and U2 corresponding to the two concepts). Then, according to formula (2), the inner product (Pq)·(Pg) of the two projection vectors obtained after projecting the two feature vectors F using the projection matrix P is calculated as the contribution Sc of the concept to be evaluated to the similarity between the two images.

[0162] FIG. 22 shows how each feature vector q, g is projected onto the plane (the plane spanned by the two concept vectors U1, U2) using such a projection matrix P (an orthogonal projection matrix onto the subspace (plane) spanned by the two concept vectors U1, U2). For simplicity of illustration, FIG. 22 shows a case where the concept vectors U1, U2 are parallel to the plane z=0 (xy plane). In other words, it shows a case where the plane spanned by the two concept vectors U1, U2 is the plane represented by z=0.

[0163] 22, the vector q is transformed into the vector q12 and the vector g is transformed into the vector g12 by the orthogonal projection matrix P calculated using the two concept vectors U1 and U2. The transformed vectors q12 and g12 are indicated by thick dashed lines.

[0164] In this case, the dot product of both vectors q12 and g12 after transformation by the orthogonal projection matrix P (in other words, the similarity of both vectors q12 and g12 in the specific subspace (plane z=0)) is calculated as the similarity Sc. In other words, when two concepts are to be considered (evaluated), the dot product (q12 g12) of the two vectors after projective transformation of two feature vectors F onto the plane spanned by the two corresponding concept vectors U1 and U2 is calculated as the similarity Sc.

[0165] In this way, the contributions of the two concept vectors U1 and U2 (in other words, the contributions of the two selected concepts to the similarity between the two images) can be calculated as the similarity Sc based on equation (2).

[0166] Similarly, the contribution (similarity) Sc of any number (predetermined number) of concept vectors U (in other words, the contribution of the predetermined number of selected concepts to the similarity between both images) can be calculated based on formula (2). Note that the contribution Sc of three or more concept vectors U can be considered as the inner product of two vectors projected from a pre-transformation space of four or more dimensions to a post-transformation space of lower dimensions (dimensions equal to the number of concept vectors) (however, this is difficult to illustrate).

[0167] <1-10. Processing of Subphase PH3b (Step S30)> <Outline of Sub-Phase PH3b> Next, the processing of sub-phase PH3b (step S30) will be described with reference to Fig. 5. The processing of sub-phase PH3b is processing such as deriving a basis for determining the similarity between two images.

[0168] In sub-phase PH3b, the image processing device 30 executes a process for explaining the basis for determining that the images are the same person (or similar people). Here, the process for deriving the basis for determining the similarity between two images (query image Xq and gallery image Xg) determined in step S12 to be images of the same person (images similar to or more than a predetermined degree) will be mainly described. However, the present invention is not limited to this, and a process for determining the similarity between any two images and a process for deriving the basis for this determination may be executed.

[0169] The processing of this subphase PH3b can analyze whether the basis for determining similarity is, for example, whether the person is wearing a shirt with a specific feature, or whether the person is wearing bottoms with a specific feature, etc. It can also analyze whether a feature was found in the pattern of the shirt, etc.

[0170] In this sub-phase PH3b, the above-mentioned concept vector U (see step S23 (FIG. 4)) is used to analyze the basis for determining similarity.

[0171] <Step S31: Extraction of key concepts> Specifically, first, in step S31 (FIG. 5), among the various concepts acquired by the learning model 400, concepts that have a particularly large influence on the similarity between two images (concepts with a large contribution) are extracted as main concepts. More specifically, among the multiple concepts, the top several concepts according to a predetermined criterion are extracted as main concepts. For example, one of the following two methods, specifically, one of the first and second methods, is executed to extract the main concepts.

[0172] Below, two methods (first method and second method) will be illustrated.

[0173] First, the first method will be described.

[0174] The first method is a method in which the contribution Sc (see formula (5)) of each concept to the similarity of an image pair is calculated for all concepts, and the top few concepts with the highest contribution Sc are extracted as main concepts.

[0175] Specifically, in the first technique, the controller 31 first calculates the contribution Sc of a single concept vector U (similarity Sc based on the above-described formula (5)) for each of a plurality of candidate concept vectors U (described later) (e.g., all concept vectors U). That is, the controller 31 executes a process for calculating the degree to which one concept represented by a single concept vector U corresponding to a certain cluster contributes to the overall similarity (a process for calculating the contribution Sc for each concept) for all concepts. Next, the controller 31 sorts all concepts in descending order of their contribution. Then, the controller 31 determines the top few concepts as main concepts (main concepts) that significantly contribute to the similarity between the two images.

[0176] In the first method (and the second method), all of the multiple concepts (multiple concept vectors U) extracted in step S20 may be identified as candidate concepts (also referred to as candidate concepts) that serve as the basis for similarity. However, the present invention is not limited to this, and the candidate concepts may be some of all of the concepts extracted in step S20. In other words, the candidate concept vectors (also referred to as candidate concept vectors U) that serve as the basis for similarity may be some of all of the concept vectors extracted in step S20. In this way, a main concept (main concept vector) may be determined from some of the candidate concepts (candidate concept vectors).

[0177] Next, the second method will be described.

[0178] The second method is similar to the first method in that it considers the degree of contribution (similarity).

[0179] However, in the second method, a search process is repeatedly performed to search for a next-ranked concept from among concepts (unselected concepts) other than concepts (selected concepts) that have already been selected (considered) at a certain point in time (at the time of the i-th iteration). Here, the next-ranked concept is a concept that, among the unselected concepts, maximizes the similarity components to be considered (similarity components additionally explained by that concept). Specifically, a search process is repeatedly performed to search for a concept vector U that maximizes the similarity components to be considered from among the concept vectors U (unselected concept vectors U) of unselected (unconsidered) concepts at a certain point in time. In other words, an unselected concept (next-ranked concept) that maximizes the newly (additionally) explainable components among the similarity components not yet explained by the selected concepts is sequentially searched for.

[0180] Specifically, a concept vector U that contributes most to the similarity is searched for in the "orthogonal complement" of the subspace spanned by the concept vector U already selected at a certain point in time (see equation (6) and the like, which will be described later). Then, the main concept is determined by repeating this search process. In this respect, the second method differs from the first method, which determines the main concept mainly by using the contribution of each concept.

[0181] The second method makes it possible to extract concepts with high independence as main concepts. In other words, it is possible to extract main concepts with little overlap between concepts.

[0182] The second method will be described in detail below.

[0183] In the second method, each feature vector F (specifically, feature vectors q and g) is transformed into a projection vector after projective transformation using a specific projection matrix R (described next).

[0184] A specific projection matrix R is a matrix (R=1−P) expressed as (1−P) using a specific projection matrix P.

[0185] A specific projection matrix P (more specifically, also denoted as Pn) is a projection matrix (more specifically, an orthogonal projection matrix) based on n (γ) concept vectors U that have already been considered at a certain point in time. The projection matrix P (more specifically, Pn) is a matrix calculated using equation (3) or (4) based on matrix B in which n concept vectors U are arranged horizontally. The value n increases by one each time the above search process is repeated.

[0186] On the other hand, the projection matrix R (more specifically, also referred to as Rn) is a matrix that projects each feature vector F onto a subspace (orthogonal complement) that is orthogonal to the projection space (subspace) defined by the projection matrix Pn, and is a matrix equivalent to (1-Pn). In other words, the orthogonal complement of the projection space (subspace) defined by the projection matrix Pn is the projection space (subspace) defined by the projection matrix Rn. The projection matrix Rn is also an orthogonal projection matrix.

[0187] The feature vector F is separated into a vector (PF) obtained by applying a projection matrix P (=Pn) that projects F onto a specific subspace, and a vector (RF) obtained by applying a projection matrix R (=Rn=1-Pn) that projects F onto the orthogonal complement of the specific subspace. The vector (PF) can also be expressed as a component that is considered in the n concept vectors U that make up the projection matrix P (specifically, matrix B). The vector (RF) can also be expressed as a component that has not yet been considered in the n concept vectors U that make up the projection matrix P (matrix B).

[0188] Furthermore, if the projection matrix P is a projection matrix onto a subspace (specific straight line) spanned by a single concept vector U, the projection matrix R is a projection matrix onto a subspace (remaining subspace) orthogonal to the subspace (specific straight line). For simplicity, if the concept vector U is a three-dimensional vector, the projection matrix R is a projection matrix onto a subspace ("plane (plane perpendicular to the specific straight line)") orthogonal to the subspace (specific straight line).

[0189] In this second method, a new concept (more specifically, a new concept vector Ur corresponding to the new concept) that maximizes the index value Qn expressed by the following formula (6) is searched for. Specifically, a new concept vector U of the (n+1)th rank is searched for. That is, the next-ranked concept vector U (and thus the next-ranked concept) is searched for.

[0190] Specifically, for each of the (remaining) concept vectors Ur (candidates for new concepts to be selected), an index value Qn is calculated based on equation (6). Here, "concept vectors already selected (considered)" refers to the n (one or more) concept vectors U used to configure the matrix Pn (the matrix corresponding to the matrix Rn). Also, "concept vectors not yet selected (considered)" refers to the concept vectors U (remaining concept vectors U) other than the n concept vectors U used to configure the matrix Pn.

[0191]

number

[0192] The vector Ur in equation (6) is an unselected concept vector U (one candidate concept vector U) corresponding to a candidate for a new concept to be selected.

[0193] Vector (Rq) is a vector obtained by applying projection matrix R to feature vector q (from the left), and vector Rg is a vector obtained by applying projection matrix R to feature vector g (from the left).

[0194] In other words, vector (Rq) is a vector obtained by projecting feature vector q onto the orthogonal complementary space of the projected space by projection matrix P, and vector (Rg) is a vector obtained by projecting feature vector g onto the orthogonal complementary space of the projected space by projection matrix P.

[0195] These vectors (Rq) and (Rg) are projection vectors onto a subspace that has not yet been considered. In addition, the dot product (Rq·Ur) between vector (Rq) and vector Ur (a new concept vector that has not yet been considered) represents the contribution component (projection component onto vector Ur) of the projection vector onto a subspace that has not yet been considered, which is contributed by vector Ur. The same is true for the dot product (Rg·b) between vector (Rq) and vector Ur (a new concept vector that has not yet been considered).

[0196] In equation (6), the product (scalar (value)) of these two dot products is calculated as the index value Qn. Specifically, the dot product ((Rq)·Ur) of vector (Rq) and vector Ur is calculated, and the dot product ((Rg)·Ur) of vector (Rg) and vector Ur is calculated, and the product (scalar product) of these dot products (the right side of equation (6)) is calculated.

[0197] This index value Qn is the product of the inner product ((RF)·Ur) between a projection vector (vector (RF) obtained by projecting feature vector F onto the orthogonal complement) of a subspace not yet considered in the already considered concepts (the "orthogonal complement" of the subspace spanned by the already considered concept vector U) and a new concept vector Ur.

[0198] Each projection vector (RF) obtained by projecting each feature vector F onto a subspace (orthogonal complementary space) that has not yet been considered by the previously considered concepts (one or more concept vectors U) represents an unconsidered component (a component that has not yet been considered by the previously considered concepts) of each feature vector F. The inner product of the unconsidered component and the new concept vector U represents the similarity between the unconsidered component and the new concept (corresponding to the new concept vector U). Therefore, the index value Qn represents the similarity between the unconsidered component of the two feature vectors q and g and the new concept. In other words, the index value Qn indicates the magnitude of the component that can be explained by the new concept in determining the similarity. In other words, the index value Qn represents the degree to which the new concept (concept vector U) contributes to the unconsidered component.

[0199] Therefore, the concept vector U that maximizes the index value Qn is the concept that maximizes the similarity with the unconsidered components regarding the two feature vectors F, and is the concept that can best explain the unconsidered components.

[0200] As a result of the search process, the concept vector Ur that maximizes the index value Qn among the unselected concept vectors Ur is determined as a new concept vector U (next-ranked concept vector). That is, the concept corresponding to the concept vector U that maximizes the index value Qn is selected as a new main concept (main concept of the next rank ((n+1)th rank)). In other words, the cluster that maximizes the index value Qn is determined as the next-ranked cluster.

[0201] Furthermore, new matrices P and R are obtained using the new concept vector (next-ranked concept vector). Specifically, with the addition of the next-ranked concept vector, the number of selected concept vectors U increases (increments) by one, and the value n also increases by one. Furthermore, in response to the increment in the number of selected concept vectors U, the rank (order) of the projection matrix P increases by one, and the rank (order) of the projection matrix R decreases by one. In other words, the number of dimensions of the projection space formed by the projection matrix P increases by one, and the number of dimensions of the projection space formed by the projection matrix R decreases by one.

[0202] Thereafter, the search process is repeated in the same manner until a predetermined number (for example, five) of main concepts are selected, thereby determining the top few (top predetermined number) main concepts (more specifically, concept vectors U corresponding to the main concepts) in the second method.

[0203] More specifically, when determining the first-ranked concept vector U, it is sufficient to start from a state where zero concept vectors U have been selected. In this case, in equation (6), it is sufficient to find the inner product of vector (Rq) and candidate concept vector U, assuming that vector (Rg) is feature vector q itself and vector (Rg) is feature vector g itself.

[0204] Alternatively, similarly to the first method, similarities Sc for multiple concept vectors U may be calculated based on formula (5), and the concept vector U corresponding to the maximum value among the calculated similarities Sc may be determined as the first-ranked concept vector U. Note that formula (6) and formula (5) are equivalent when determining the first-ranked concept (Rq·Ur=q·Ur=qs=Pq).

[0205] Furthermore, for example, after the first-ranked concept vector U is determined as the concept vector U110, when the second-ranked concept vector U is determined, the matrix B becomes a single concept vector U110 (column vector). Then, a matrix P (rank = 1) is calculated based on the matrix B (column vector) (see formula (3) or (4)), and a matrix R is calculated based on the matrix P (R = 1 - P). Then, a next-ranked (second-ranked) concept vector U (for example, concept vector U400) that maximizes the index value Qn of formula (6) is calculated.

[0206] Furthermore, when the third-ranked concept vector U is determined, matrix B is a matrix in which two concept vectors U110 and U400 are arranged horizontally. Then, matrix P (rank = 2) is calculated based on matrix B (see equation (3) or (4)), and matrix R is calculated based on matrix P (R = 1 - P). Then, the next-ranked (third-ranked) concept vector U (e.g., concept vector U510) that maximizes the index value Qn of equation (6) is calculated.

[0207] Thereafter, the same process is repeated, and the top few (for example, five) concept vectors U are determined.

[0208] <Another end condition> Here, the search process is repeated until a predetermined number of main concepts are obtained (in other words, the termination condition is set to be that a predetermined number of main concepts are obtained), but this is not limited to this, and the search process may be repeated under a different termination condition (termination determination condition).

[0209] For example, the termination condition may be set to satisfy the condition of equation (7). Specifically, the termination condition of equation (7) may be determined after the value n is incremented.

[0210]

number

[0211] The left side of equation (7), ((Pq)·(Pg)), is the same as the similarity Sc described above (see equation (2)). The matrix P is a projection matrix onto a subspace spanned by n selected concept vectors U, which represent selected concepts (already selected concepts). In other words, the left side of equation (7) represents the inner product of the projection vectors obtained by projecting vectors q and g onto the subspace (i.e., the contribution of the n selected concept vectors U).

[0212] On the other hand, the right side of equation (7) is the value obtained by multiplying the inner product of the two vectors q and g before transformation by a constant ratio (1-δ), where δ is a constant (0<δ<1) that indicates the ratio, and (1-δ) also indicates the ratio to the whole.

[0213] For example, when the value δ is 0.4, the value (1-δ) is 0.6. In this case, the termination condition of equation (7) means that the inner product of both transformed vectors (Pq) and (Pg) is greater than 60% of the inner product of both pre-transformed vectors q and g. In other words, equation (7) means that the search process ends when the value Sc (the similarity Sc calculated using the projection matrix P corresponding to n concept vectors U), which gradually increases with the increase in value n, becomes greater than a certain percentage (1-δ) of the total similarity St.

[0214] For example, after the first five concept vectors U have been determined, when searching for a new (n+1)th (specifically, sixth) concept vector Ur, if the termination condition of equation (7) is met, the repeated search process ends. Then, the first five concept vectors U are determined as the main concept vectors.

[0215] In this way, instead of determining the number n of concept vectors U in advance, the value δ (and thus the proportion of the contribution to be considered among the n concept vectors U) may be determined in advance.

[0216] <Example of concept decision> According to the second method described above, for example, the top (first) concept C1, the second concept C2, the third concept C3, the fourth concept C4, and the fifth concept C5 are searched for and found in this order.

[0217] More specifically, the concept vector U110 (see the top row of FIG. 18) is identified as the top concept C1 (see also FIGS. 11 and 13). In this case, it can be seen that the most important basis for determining the similarity between the two images (the basis for determination by the trained model 420) is the feature expressed by the concept vector U110. In other words, it can be seen that the feature of the cluster G110 corresponding to the concept vector U110 is the most important basis for determining (the similarity). Since the cluster G110 is composed of a group of images of a person wearing a "white shirt with many patterns" (see FIGS. 11 and 13), the concept C1 can also be expressed as a "white shirt with many patterns."

[0218] Furthermore, concept vector U400 (see the top row of Figure 20) is identified as the second-ranked concept C2. In other words, it can be seen that the characteristics of cluster G400 corresponding to concept vector U400 are a relatively large basis for judgment from a different perspective than the first-ranked concept C1. Since cluster G400 is composed of a group of images of people wearing "short white bottoms" (see Figure 11, etc.), concept C2 can also be expressed as "short white bottoms."

[0219] Similarly, concept vector U510 (see middle row of Figure 20) is identified as the third-ranked concept C3 ("light blue bottoms"), concept vector U200 (see top row of Figure 19) is identified as the fourth-ranked concept C4 ("light pink shirt"), and concept vector U310 (see middle row of Figure 19) is identified as the fifth-ranked concept C5 ("plain wine-red shirt").

[0220] <Step S32: Explanation of main concepts, etc.> Next, in step S32 (FIG. 5), a process is executed to display the analysis results, etc., from step S31. In other words, a process is executed to present the basis (explanatory information) for determining that the two images are similar to each other.

[0221] 25 is a diagram showing an example (screen display example) of the search processing result (analysis processing result) described above. Note that the display screen 600 of FIG. 25 and various other screens 610, 620 (described later) are displayed on the display unit 35b, for example.

[0222] On the display screen 600 of FIG. 25, five main concepts C1 to C5 based on the second method are presented.

[0223] Specifically, in a graph display area 609 of the display screen 600, the similarity Sc calculated based on Equation (2) (more specifically, Equation (5)) is displayed in a graph form. For the top few (here, five) concepts, the contributions to the similarity between the image pairs (here, the similarity between the query image and the gallery image) are calculated, and the contributions are displayed. The contributions may be displayed as a numerical value (e.g., 0.18) or may be displayed as a graph (with the numerical values ​​converted into the lengths of the bar graphs) as shown in FIG. 25.

[0224] Here, five main concepts C1 to C5 ranked first to fifth are determined based on the second method. The similarity (contribution) Sc is also shown for each of these five main concepts C1 to C5. Furthermore, the contributions (remaining contributions) of concepts other than the five main concepts C1 to C5 are also shown (in the "Other" column). The remaining contributions are calculated, for example, by subtracting the similarity Sc (see formula (2)) calculated using the five concepts C1 to C5 as the selected concepts from the overall contribution St (see formula (1)).

[0225] In particular, the five concepts C1 to C5 are extracted as main concepts so that they are arranged in descending order from highest to lowest. This order C1 to C5 is based on the second method and is different from the order based on the similarity Sc calculated by formula (5) (the order based on the first method).

[0226] If the first method is used, the five concepts are extracted in the order of C1, C2, C4, C3, and C5 (descending order of similarity for each concept) based on the similarity Sc calculated by formula (5). In contrast, the second method extracts the concepts in the order of C1, C2, C3, C4, and C5. That is, even though the similarity Sc for concept C3 according to formula (5) is smaller than the similarity Sc for concept C4 according to formula (5), concept C3 is extracted as a higher-ranking concept than concept C4.

[0227] Furthermore, in the first method, based on the similarity Sc calculated by Equation (5), lower-level concepts (lower-level concept vectors) of the first-ranked concept C1 (concept vector U110 corresponding to cluster G110) may also be extracted as relatively higher-level concepts. Specifically, not only the concept vector U110 (representing cluster G110) but also its lower-level concept vectors U111 and U112 (representing clusters G111 and G112 (see FIG. 11)) may be extracted as major concept vectors (concepts). This is because if concept vector U111 is similar to concept vector U110, the similarity Sc for concept vector U111 may also be as high as that for concept vector U110. Note that concept vector U111 is a concept vector representing lower-level cluster G111 of cluster G110, and concept vector U112 is a concept vector representing lower-level cluster G112 of cluster G110.

[0228] On the other hand, in the second method, as described above, a concept vector that maximizes the similarity component to be considered is searched for among concept vectors other than the concept vector already selected (considered) at a certain point in time (e.g., concept vector U110). As a result, other concept vectors U that are relatively different (highly independent) from concept vector U110 are more likely to be extracted as major concept vectors (major concepts). Conversely, concept vectors U111 and U112 representing lower clusters G111 and G112 of cluster G110, respectively, are less likely to be extracted as major concept vectors (major concepts). In the example of FIG. 25, concept vectors U111 and U112 are not extracted as the top five major concept vectors (major concepts).

[0229] Thus, the second method makes it possible to extract concepts with high independence as main concepts. In other words, it is possible to extract main concepts with minimal overlap between concepts.

[0230] 25 also has buttons 601 to 605 and the like in addition to a graph display area 609. Each of the buttons 601 to 605 displays an identifier (such as "G110") of the corresponding cluster G. The image processing device 30 and the user can uniquely specify (identify) each corresponding cluster of each of the concepts C1 to C5 and the like using the identifier (identification ID).

[0231] Further, detailed information about each concept is displayed as follows.

[0232] When the text portion of each concept C1 to C5 in the graph display area 609 of the display screen 600 or the buttons 601 to 605 provided directly below each text portion is pressed by operating the mouse or the like, a detailed information screen regarding the corresponding concept is displayed.

[0233] For example, when button 601 immediately below concept C1 is pressed, detailed information screen 610 (see FIG. 26) is displayed. Also, when button 602 immediately below concept C1 is pressed, detailed information screen 620 (see FIG. 27) is displayed. The same applies to the other buttons.

[0234] Fig. 26 is a diagram showing a display screen 610 for detailed information about concept C1. Fig. 27 is a diagram showing a display screen 620 for detailed information about concept C2. Similarly, display screens for detailed information (also referred to as detailed information screens) also exist for other concepts C3 to C5, etc. The following description will focus on the display screen 610 for detailed information about concept C1.

[0235] The display screen 610 has areas 611 to 614 and a button 615 .

[0236] The area 611 (also referred to as the upper area) is a display area for a plurality of images (concept constituent images) that make up the corresponding cluster G110 of the concept C1.

[0237] Area 612 (also referred to as the lower area) is a diagram showing a heat map image indicating areas (ignition areas) determined to be characteristic areas by the trained model 420 for each of the multiple images displayed in area 611. In area 612, a heat map image corresponding to each image is displayed directly below each image in the upper area 611. From the heat map in the lower area 612, it can be seen that the trained model 420 is focusing on the shirt part of each person.

[0238] The area 613 is a display area for a "concept visualization image" (described later). The concept visualization image is an image that visualizes the concept (concept vector) of the cluster G110. The concept visualization image is a virtual input image that corresponds to the representative vector (concept vector) of the cluster G110, and is generated using a feature visualization method (described later) or the like. The concept visualization image can also be expressed as an image that expresses the abstract concept (abstract concept) of the cluster G110.

[0239] The area 614 is a display area for attribute information (concept name, etc.) of the cluster G110. As will be described later, when the image processing device 30 receives an operation input (input of attribute information of the cluster (character input, etc.)) from the user, the image processing device 30 stores the attribute information in the storage unit 32 and displays the attribute information (concept name, etc.) in the area 614. Note that FIG. 26 shows a display state after the operation input (character string "CONCEPT C1: WHITE SHIRT WITH LOTS OF PATTERNS"). Before the operation input (before the character string "WHITE SHIRT WITH LOTS OF PATTERNS" is input), for example, only the character string "CONCEPT C1" is displayed in the area 614.

[0240] Button 615 is a button that accepts an instruction to display information about related clusters. When button 615 is pressed, a dendrogram like the one shown in the upper part of FIG. 11 (particularly around concept C1) and / or a Venn diagram like the one shown in the lower part of FIG. 11 (or FIG. 13) are displayed. This displays the existence and inclusion relationships of related clusters (specifically, same-level clusters G120 and G200, higher-level cluster G100, lower-level clusters G111 and G112, etc.) with respect to cluster G110 corresponding to concept C1. Furthermore, by clicking the corresponding position of each related cluster in the dendrogram or Venn diagram with the mouse, a screen displaying detailed information about the related cluster (e.g., same-level cluster G120) is displayed. Note that this is not a limitation, and a display screen like that shown in FIG. 13 may also be displayed in response to the mouse click. In detail, on the display screen, each group of images that constitutes the related clusters of cluster G110 (G110, G120, G100, G200, etc.) may be displayed together with a Venn diagram (a diagram showing the hierarchical relationships (inclusion relationships) of related clusters) near cluster G110.

[0241] The user can obtain detailed information about the concept C1 from the display screen 610 of FIG.

[0242] Specifically, it can be seen from the heat map of the lower region 612 that the trained model 420 is focusing on the shirt portion of each person.

[0243] Furthermore, the multiple images in the upper area 611 make it possible to visually understand what images constitute the concept C1 (more specifically, its corresponding cluster).

[0244] Furthermore, the concept visualization image in area 613 allows one to obtain an image that abstractly visualizes concept C1.

[0245] Furthermore, the attribute information in the area 614 (specifically, the concept name and user remarks, etc.) allows the content of the concept to be known through linguistic expression.

[0246] Here, the concept name of the cluster G110 may be grasped and input by the user in the following manner: The same applies to the concept names of the other clusters.

[0247] Specifically, first, the user visually recognizes a detailed information screen 610 (FIG. 26) related to the cluster G110. The detailed information screen 610 (more specifically, its upper area 611) displays a group of images (see also FIG. 13) that make up the cluster G110.

[0248] Thereafter, the user presses button 615 to display a Venn diagram like the one shown in the lower part of Fig. 11. Then, while referring to the Venn diagram, the user presses the position of cluster G120 in the Venn diagram to display a detailed information screen (detailed information screen similar to Fig. 26) regarding cluster G120 of the same level as cluster G110. The detailed information screen regarding cluster G120 displays the group of images that make up cluster G120 (see Fig. 13).

[0249] Furthermore, if necessary, a detailed information screen relating to the upper cluster G100 of the cluster G110 can also be displayed by a similar operation.

[0250] The user can understand the characteristics of each cluster by comparing the images included in these detailed information screens.

[0251] For example, by comparing cluster G110 with the same cluster G120, the characteristics of both clusters G110 and G120 can be understood. Specifically, it can be seen that cluster G120 is composed of images of a person wearing a "white shirt with little pattern," and cluster G110 is composed of images of a person wearing a "white shirt with a lot of pattern."

[0252] Furthermore, since the upper cluster G100 is a cluster that includes both clusters G110 and G120, it is clear that it is a "white shirt with a pattern."

[0253] Furthermore, by comparing the higher-ranking cluster G100 with its peer cluster G200, the characteristics of both clusters G100 and G200 can be understood. Specifically, it can be seen that cluster G100 is composed of images of people wearing "patterned white shirts," while cluster G200 is composed of images of people wearing "light pink shirts." In other words, it can be seen that cluster G100 is "a shirt other than light pink."

[0254] Through these considerations, the user can determine "heavily patterned white shirt" as the concept name of cluster G110. Then, the user inputs the character string "heavily patterned white shirt" into area 614 on the detailed information screen 610. In response to this, the image processing device 30 accepts this input operation and registers "heavily patterned white shirt" in the storage unit 32 as the concept name of cluster G110. Furthermore, when the detailed information screen 610 is subsequently displayed, the concept name "heavily patterned white shirt" is displayed in area 614.

[0255] However, without being limited to this, the user may acquire all of this information by visually checking a display screen such as that shown in FIG. 13, and grasp the concept name of the cluster G11.

[0256] Furthermore, such grasping and input of concept names may be performed at this time (step S32) for only the cluster desired by the user and its related clusters (i.e., only a relatively small number of clusters). However, this is not limitative, and the process may be performed for all clusters immediately after step S23.

[0257] <Concept visualization image> Here, the process of generating a concept visualization image using the Feature Visualization method will be described.

[0258] FIG. 28 is a diagram showing an outline of the process of generating a concept visualization image using the Feature Visualization method. The Feature Visualization method is a technique for investigating what input causes a specific firing (a specific intermediate output) in a certain intermediate layer of a neural network. Here, the final output from the output layer (i.e., feature vector F) is used instead of the intermediate output from the intermediate layer. In other words, it is investigated what input image causes the output vector (feature vector F) from the trained model 420 to occur.

[0259] Specifically, as shown in FIG. 28, an input image that minimizes the distance d(F,U) between the output vector (i.e., feature vector F) output from the trained model 420 and the target vector (here, a certain concept vector U) is obtained as the concept visualization image. The feature vector F is a function (F(I)) of the image vector I of the input image. Note that here, the input image is represented by the image vector I (a vector obtained by rearranging pixel values ​​that were arranged two-dimensionally in the input image into one dimension). Also, the distance d(F,U) etc. will be abbreviated as simply the distance d etc.

[0260] More specifically, the process of finding the increment δI that minimizes the increment δd of this distance d using the internal parameters of the trained model 420 (acquired by machine learning) is repeated, and the input image vector I (concept visualization image that reflects the characteristics of the concept vector U) is found.

[0261] The increment δd of the distance d is expressed by the following equation (8). Here, vector H is a constant vector (a vector with each component being a constant) calculated using the weights and activation functions of each hidden layer of the neural network. Note that equation (8) is derived by, for example, summing up the relationship (expressed by gradient information of the trained model 420, etc.) between the increment δIk of the value Ik (scalar), which is the k-th component of the image vector I, and the increment δd of the distance d(F(Ik),U) for all components k. This distance d(F(Ik),U) is the distance between the feature vector F(Ik) corresponding to the image having the k-th component Ik and the concept vector U.

[0262]

number

[0263] As shown in equation (8), the increment δd is expressed as the inner product of vector H and vector δI (the increment vector of vector I).

[0264] Furthermore, the process of minimizing the distance d is realized by repeatedly setting the increment δd to the smallest value (the norm of which is the most negative value (also referred to as the maximum negative value)). More specifically, this process is realized by repeatedly finding a δI that minimizes the increment δd among δIs having the same norm (magnitude). This process is equivalent to repeatedly finding the orientation of δI that can minimize the distance d (minimize the increment δd).

[0265] The direction of δI that minimizes the increment δd is the direction that minimizes the dot product with vector H (i.e., makes it the negative value of the maximum norm). Therefore, δI that minimizes the increment δd (makes it the negative value of the maximum norm) can be found as the vector (δI when cosθ=-1) that has a predetermined norm (magnitude) and is oriented in the opposite direction to the direction of the constant vector H. Here, the angle θ is the angle between both vectors H and δI.

[0266] Then, the operation of adding such vector δI to vector I (see the following equation (9)) is repeated many times (for example, thousands of times). Note that the initial value (initial vector) of vector I may be a vector corresponding to a random noise image (or an appropriate input image (such as an input image belonging to a cluster corresponding to concept vector U)). Also, ε is a predetermined constant.

[0267]

number

[0268] By this process, an input image vector I (that is, an input image obtained by rearranging the input image vector I into a two-dimensional array) that minimizes the distance d between the feature vector F and the target vector (concept vector U) is obtained.

[0269] For example, the "concept visualization image" of concept C2 (see area 623 in FIG. 27) is an image generated using the Feature Visualization method based on the concept vector U400 (see the top row in FIG. 20) of cluster G400 ("short white bottoms"). This concept visualization image has a part that resembles "short white bottoms (white shorts)" near the center of the image (near the area indicated by the thick dashed circle). This image shows that the concept vector U400 reflects the feature of "short white bottoms."

[0270] Furthermore, the "concept visualization image" of concept C1 (see area 613 in FIG. 26) is an image generated using the Feature Visualization method based on the concept vector U110 (see the top row in FIG. 18) of cluster G110 ("white shirt with many patterns"). This concept visualization image has a part that resembles a "patterned shirt" in a location slightly above the center of the image. This image shows that the concept vector U400 reflects the feature of a "patterned (heavily patterned) shirt."

[0271] It is not necessarily easy for a user to fully grasp the content of each concept from only the "concept visualization image" of each concept, so it is preferable that the concept visualization image is used as an auxiliary image.

[0272] <1-11. Effects of the embodiment> According to the above embodiment, a plurality of hierarchical clusters are generated by performing a hierarchical clustering process on a plurality of feature vectors F (step S22 (FIG. 4)). Then, a vector (specifically, a concept vector U) corresponding to a specific cluster among the plurality of clusters is extracted as a concept of the specific cluster (step S23).

[0273] Therefore, it is possible to manage and understand the concepts corresponding to specific hierarchical clusters using their representative vectors (concept vectors U). Furthermore, by performing analysis using the concept vectors U, it becomes possible to explain the basis for image similarity on a concept basis.

[0274] Furthermore, according to the above embodiment, two or more input images corresponding to a specific cluster (e.g., G110) among the multiple clusters generated by the hierarchical clustering process are determined and displayed as an image group representing the concept of the specific cluster (specific concept) (see, e.g., FIGS. 13 and 26). In other words, two or more input images constituting the specific cluster are displayed as an image group corresponding to the concept of the specific cluster. For example, multiple input images corresponding to cluster G110 (see area 611 in FIG. 26) are displayed as an image group corresponding to the concept of cluster G110. This image group is a group of images similar to each other (similar image group) and represents the concept of the specific cluster, and is therefore also referred to as a concept-representing similar image group. This makes it possible to visually present to the user the concept of the specific cluster.

[0275] Furthermore, according to the above embodiment, a group of similar images in a specific cluster is displayed together with the respective heat maps (see area 612 (FIG. 26) etc.). Therefore, it is possible to identify ignition regions (feature regions) in images related to a specific cluster using the heat map, and then conceptually grasp the features of the specific cluster.

[0276] Furthermore, a group of similar images of a specific cluster is displayed together with a concept visualization image (see area 613 (FIG. 26), etc.). Therefore, it is possible to grasp the characteristics of a specific cluster conceptually (especially logically) through the group of similar images, while visually grasping the characteristics of the specific cluster through the concept visualization image.

[0277] In the above embodiment, a plurality of concepts (more specifically, a plurality of concept vectors U) corresponding to a plurality of hierarchical clusters are extracted. Therefore, it is possible to grasp the superordinate / subordinate relationships (parent-child relationships) and inclusion relationships between the plurality of concepts.

[0278] Furthermore, according to the above embodiment, when determining the similarity between two images (first and second images) in step S31 (FIG. 5), the contribution of multiple candidate concepts to the similarity between the first and second images is calculated, making it possible to grasp the contribution (importance) of each concept to the similarity between the image pair.

[0279] In the above embodiment, the contribution degree is calculated for each of a large number of concepts (candidate concepts), and the top few main concepts are determined by sorting based on the contribution degree. However, this is not limited to this. For example, the contribution degree to the similarity between two images may be calculated only for at least one concept (designated concept) designated by the user (based on the user's interests, etc.). In particular, the contribution degree to the similarity between two images may be calculated only for the concept vector U (designated concept vector) corresponding to the designated concept. This allows the user to know to what extent the designated concept designated arbitrarily contributes to (influences) the similarity between two images.

[0280] In the above embodiment, the basis for the similarity between two images (first image and second image) is displayed by the contribution (numerical value) of a predetermined number of top concepts (one to several) as shown in Fig. 25. Therefore, it is possible to provide an objective evaluation standard regarding the basis for judgment.

[0281] In particular, the inner product of two projection vectors obtained by projecting two feature vectors F corresponding to two images onto a subspace (a line) spanned by the concept vector U specific to each concept is calculated as the contribution Sc of each concept (see formula (5)). Therefore, the contribution of each concept to the similarity can be objectively presented.

[0282] 25 shows that, among a plurality of hierarchical concepts, the contribution Sc of concept C1 is higher than the contribution Sc of concept C2. This shows that, in the similarity determination by the image processing device 30, the feature of concept C1 ("white shirt with many patterns") contributes more than the feature of concept C2 ("white, short bottoms").

[0283] In other words, it can be seen that the "similarity judgment" (the judgment that two images are similar to each other) is based on the fact that the person is wearing a "shirt" with specific characteristics rather than the fact that the person is wearing "bottoms" with specific characteristics.

[0284] Furthermore, in FIG. 25, concept C1 (cluster G110) is determined to be the concept with the highest contribution Sc. This also indicates that the contribution Sc of cluster G110 is greater than the contribution Sc of its peer cluster G120 (see FIG. 11) (see the left-pointing arrow in FIG. 12). That is, the feature of cluster G110 ("white shirt with many patterns") contributes more than the feature of cluster G120 ("white shirt with few patterns"). In other words, it can be seen that similarity is judged based on the feature of "(white shirt with many patterns)" rather than the feature of "(white shirt with few patterns)." In other words, it can be seen that the "similarity judgment" is made by finding a particular feature in the fact that the shirts have many patterns.

[0285] Furthermore, the fact that concept C1 (cluster G110) has the highest contribution indicates that the contribution Sc of cluster G110 is greater than the contribution Sc of its superordinate cluster G100 (see FIG. 11) (see the downward arrow in FIG. 12). That is, it indicates that the feature of cluster G110 ("white shirt with many patterns"), which is a more detailed feature (lower-level feature) than the feature of cluster G100 ("white shirt with many patterns"), contributes greatly. In other words, it can be seen that similarity is judged not simply based on "white shirt with many patterns" but also based on "heavy patterns."

[0286] Furthermore, the fact that concept C1 (cluster G110) has the highest contribution also indicates that the contribution Sc of cluster G110 is greater than the contributions Sc of its lower-level clusters G111 and G112 (see FIG. 11) (see the upward arrows in FIG. 12). That is, it is shown that the feature of cluster G110 ("white shirt with a lot of patterns") contributes more than the feature of cluster G111 ("white shirt with a lot of patterns (and the pattern is linear)") (see also FIG. 13). It is also shown that the feature of cluster G110 ("white shirt with a lot of patterns") contributes more than the feature of cluster G112 ("white shirt with a lot of patterns (and the pattern is curved)"). That is, it can be seen that the similarity is determined without taking into consideration the type of pattern (straight or curved).

[0287] <1-12. Modification of the first embodiment> In the above embodiment, a vector (concept vector U) corresponding to a specific cluster is extracted as the concept of the cluster, but the present invention is not limited to this.

[0288] For example, in the first embodiment, a "subspace" corresponding to a specific cluster may be extracted as a concept of the specific cluster (a specific subspace representing the concept). In other words, a "subspace" corresponding to a specific cluster may be extracted as a concept expression of the specific cluster (a concept expression using a subspace).

[0289] In detail, a subspace spanned by the concept vector U of the specific cluster itself (specifically, a straight line (a straight line including the concept vector U) after projection by the projection matrix P corresponding to the concept vector U) may be extracted as a concept of the specific cluster. For example, a subspace spanned by the concept vector U100 (see the bottom of FIG. 18) of the cluster G100 (see the bottom of FIG. 11) itself (i.e., a straight line including the concept vector U100) may be extracted as a concept of the cluster G100.

[0290] Alternatively, a subspace spanned by a predetermined number of concept vectors U corresponding to a predetermined number (e.g., two) of lower clusters included in a specific cluster may be extracted as a concept (concept representation) of the specific cluster. For example, a subspace (plane) spanned by two concept vectors U110 and U120 (see FIG. 18) corresponding to two lower clusters G110 and G120 of cluster G100 (see the lower part of FIG. 11) may be extracted as a concept of cluster G100.

[0291] In particular, in the first embodiment, in the similarity judgment between two images, a specific concept vector U is extracted as a basis concept for judging that the two images are similar to each other ("similarity judgment"), but this is not limiting. For example, a subspace (straight line) spanned by the specific concept vector U may be extracted as a basis concept for the similarity judgment.

[0292] The same applies to the second embodiment described below. In particular, in the second embodiment, a specific concept vector U is extracted (described below) as a basis concept for determining that two images are not similar to each other ("dissimilarity determination"), but this is not limiting. For example, a subspace (straight line) spanned by the specific concept vector U may be extracted as a basis concept for the "dissimilarity determination."

[0293] 2. Second Embodiment In the first embodiment described above, a process for explaining the basis for a determination that two images are similar to each other ("similarity determination") (FIG. 5) has been exemplified as a process for generating explanatory information regarding an inference result. However, this is not limited thereto, and for example, a process for explaining the basis for a determination that two images are not similar to each other ("dissimilarity determination") (see FIG. 6) may be performed as a process for generating explanatory information regarding an inference result. In other words, the basis for "similarity" may be the basis for a dissimilarity determination, rather than the basis for a similarity determination. In the second embodiment, such an aspect will be described. The following description will focus on differences from the first embodiment. In the second embodiment, step S40 (FIG. 6) is executed instead of step S30 (FIG. 5).

[0294] FIG. 29 is a conceptual diagram showing the process (step S41) for determining the basis for the "dissimilarity judgment." In FIG. 29, for convenience of illustration, the entire space is abstracted and expressed as a three-dimensional space. The concept vector U has the same direction as the z axis, and a straight line extending in the z direction indicates the subspace spanned by the concept vector U. The xy plane (the plane where z=0) indicates the orthogonal complement of the subspace spanned by the concept vector U. Here, the two feature vectors q and g are feature vectors F corresponding to two images that are dissimilar to each other. FIG. 29 shows that the directions of the two feature vectors F are significantly different from each other, that is, the two images are dissimilar to each other.

[0295] As shown in FIG. 29, an orthogonal complementary space (here, a plane perpendicular to the concept vector U) of a subspace (straight line) indicated by a specific concept vector U corresponding to a specific concept (specific cluster) is assumed.

[0296] The projection vector (RF) (specifically, Rq, Rg) obtained by projecting the feature vector F onto the orthogonal complementary space contains components of the feature vector F excluding those explained by the concept vector U (components (residual components) that have not yet been explained by the concept vector U). The fact that two such projection vectors (RF) are close to each other means that the components (residual components) that have not yet been explained are similar (and therefore, that the residual components do not have a significant effect on the similarity judgment (dissimilarity judgment) of the two feature vectors F). Conversely, this means that the components that have already been explained by the concept vector U have a significant effect on the dissimilarity judgment of the two feature vectors F.

[0297] By utilizing such characteristics, in the second embodiment, when the distance between projection vectors Rq and Rg obtained by projecting two feature vectors q and g onto the orthogonal complementary space is small (especially very close), the specific concept (corresponding specific concept vector U) is extracted as the "basis concept for dissimilarity judgment." More specifically, a specific concept vector U for which the distance between the projection vectors Rq and Rg of the subspace (subspace (straight line) spanned by the specific concept vector U) onto the orthogonal complementary space (plane, etc.) is relatively small (compared to other concept vectors U) is extracted as the "basis concept for dissimilarity judgment." A more specific description will be given below.

[0298] When the projection matrix onto a subspace indicated by a specific concept vector U corresponding to a specific concept (specific cluster) is expressed as a matrix P (see equation (3) etc.), the projection matrix R onto the orthogonal complementary space of the subspace is expressed as (1-P). Therefore, the projection vector obtained by projecting feature vector q onto the orthogonal complementary space is the vector ((1-P)q), and the projection vector obtained by projecting feature vector g onto the orthogonal complementary space is the vector ((1-P)g) (see FIG. 29).

[0299] The distance between these projection vectors is |((1-P)q)-((1-P)g)|, and the square of this distance is defined as the evaluation value Sd1 (see equation (10)).

[0300]

number

[0301] As described above, the matrix P (see equation (3) etc.) is a matrix that differs depending on the matrix B and therefore the concept vector U. Therefore, finding the matrix P that minimizes the evaluation value Sd1 is equivalent to finding the concept vector U that minimizes the evaluation value Sd1. Therefore, the concept vector U that minimizes this evaluation value Sd1 is extracted as the "basis concept for dissimilarity judgment."

[0302] Furthermore, minimizing the evaluation value Sd1 is equivalent to "maximizing" the evaluation value Sd2 in equation 11. A simpler evaluation value Sd2 may also be used.

[0303]

number

[0304] In this way, the evaluation value Sd2 (or Sd1) is used to obtain the main concept vector U as the "basis concept for dissimilarity judgment."

[0305] In this case, a method similar to the above-mentioned first method (also referred to as the third method) or a method similar to the second method (also referred to as the fourth method) may be used.

[0306] Specifically, in the third method, the controller 31 first calculates an evaluation value Sd2 (the above-mentioned formula (11)) for a single concept vector U for each of a plurality of candidate concept vectors U. Next, the controller 31 sorts the concepts corresponding to the plurality of candidate concept vectors U in descending order of their evaluation values ​​Sd2. The controller 31 then determines the top few concepts as the "basis concepts for determining dissimilarity" between the two images (especially the main concepts).

[0307] On the other hand, in the fourth method, a search process is repeatedly executed to search for a concept that maximizes the evaluation value Sd2 (or minimizes the evaluation value Sd1) among concepts (unselected concepts) other than those already selected (considered) at a certain point in time (at the time of the iterative process). The search process is repeatedly executed until a predetermined termination condition is met (for example, until a predetermined number of main concepts are determined). Note that with each iteration, the number of selected concept vectors U increases by one, and accordingly, the rank (order) of the projection matrix P increases by one.

[0308] Each method can also calculate the evaluation value D shown in Equation (12). This evaluation value D refers to the components (residual components related to the similarity evaluation) that have not yet been considered by the n concepts (concept vector U) selected up to a certain point in time regarding the "dissimilar" judgment. Specifically, the evaluation value D is calculated as the value (also called the residual) obtained by subtracting the degree of "dissimilar" judgment (Sd2 / 2) explained by the n concepts from the degree of "dissimilar" judgment (dissimilarity judgment) (1-q·g). Note that the evaluation value Sd2 is multiplied by a coefficient 1 / 2 to adjust the evaluation value D to zero when P is a unit matrix (full rank) (a β × β unit matrix) (i.e., when the concept (concept vector U) corresponding to the entire space is considered).

[0309]

number

[0310] In the fourth method, this evaluation value D may be used (instead of the evaluation value Sd2, etc.). That is, a search process may be executed to search for a concept that minimizes the evaluation value D among concepts (unselected concepts) other than concepts already selected (considered) at a certain point in time. The search process is repeatedly executed until a predetermined termination condition is met (for example, until a predetermined number of main concepts are determined). With each repetition, the number of selected concept vectors U increases by one, and accordingly, the rank (order) of the projection matrix P increases by one, and the evaluation value D gradually decreases.

[0311] By using the third or fourth method described above, a specific concept vector U from among the multiple concept vectors U corresponding to multiple clusters is extracted as a "basis concept for dissimilarity judgment." The specific concept vector U is a concept vector that minimizes the distance between the projection vectors (Rq) and (Rg) of feature vectors q and g onto the orthogonal complementary space of the subspace corresponding to the specific concept vector U.

[0312] In other words, among the multiple subspaces corresponding to the multiple clusters, the concept vector U that spans the subspace that makes the distance between the projection vectors ((1-P)q) and ((1-P)g) of the two feature vectors q and g onto the orthogonal complementary space relatively smaller (than other subspaces) is extracted as the "basis concept for dissimilarity judgment."

[0313] In other words, among the multiple concepts corresponding to the multiple clusters, a concept that, if removed, would result in the two images being determined to be similar to each other is extracted as a "basis concept for dissimilarity judgment."

[0314] As a result of the above processing, the following "basis concept for dissimilarity judgment" can be extracted.

[0315] For example, if an image of a person wearing a light pink shirt and an image of a person wearing a plain wine-red shirt are judged to be dissimilar, concept vector U200 and concept vector U310, etc., may be extracted as the "basis concepts for the dissimilarity judgment" (see Figure 11, etc.).

[0316] Alternatively, (although not shown in FIG. 11, etc.) if there is a super-cluster of "blue shirt" and a sub-cluster of "blue and checked shirt," an image of a person wearing a blue shirt and an image of a person wearing a blue and checked shirt may be determined to be dissimilar. As the basis concept for this dissimilarity determination, the concept vector of the "blue shirt" cluster and the concept vector of the "blue and checked shirt" cluster, etc., may be extracted.

[0317] In this way, the concept C (concept vector U) that best reflects the characteristics of both images to be compared can be extracted as the main concept.

[0318] Furthermore, in step S42, a display similar to that in Figures 25 and 26 is displayed. However, a display for explaining the basis concept of the "dissimilarity judgment" is displayed instead of the basis concept of the "similarity judgment".

[0319] Specifically, for example, similar to Fig. 25, the evaluation values ​​Sd2 (or Sd1) of the top few (for example, 2 to 5) main concepts are displayed in a graph. The evaluation values ​​D (residuals) may also be displayed. Also, similar to Fig. 26 etc., detailed information about each main concept (constituent images of the corresponding cluster, heat map images, concept visualization images, etc.) is displayed.

[0320] Also, similar to FIG. 26 etc., detailed information about each of the top several main concepts (basis concepts for "dissimilarity judgment") is displayed.

[0321] According to the above process, when an image pair is dissimilar, it is possible to grasp the basis for determining that the images are dissimilar (the basis for determining that the images are dissimilar).

[0322] In the second embodiment, a specific concept vector U is extracted as the "basis concept for dissimilarity determination." However, as described above, this is not limiting, and for example, a subspace (straight line) spanned by the specific concept vector U may be extracted as the "basis concept for dissimilarity determination."

[0323] <3. Modifications, etc.> Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described contents.

[0324] For example, the processing of sub-phase PH3a (step S20) does not necessarily have to be performed after the second phase PH2 (step S12), and may be performed immediately after the first phase PH1 (step S11), for example.

[0325] Furthermore, in each of the above embodiments, in sub-phase PH3a, the multiple input images 211 used in machine learning are used as the input images 210 for the trained model 420. However, this is not limiting, and multiple input images (e.g., input images 213) other than the multiple input images 211 may be used. However, it is preferable to use the multiple input images 211 used in training the trained model 420 as the input images 210 for the trained model 420 rather than using the multiple input images (input images 213, etc.). This is because it is considered that a relatively accurate distribution is obtained as the distribution of feature vectors 250 for the multiple input images 210 (output distribution from the trained model 420).

[0326] In addition, although the above-described embodiments illustrate examples in which the present invention is applied to person recognition, the present invention is not limited to this. For example, the present invention may be applied to product recognition, lesion recognition (lesion detection), etc.

[0327] Furthermore, in each of the above embodiments, an example is given in which the present invention is applied to metric learning (distance learning), but the present invention is not limited to this, and may also be applied to class classification learning, etc.

[0328] For example, the present invention may be applied to class classification learning using a trained model 420 including a feature extraction layer (configured as a CNN or the like) that extracts image features and a fully connected layer that performs classification processing or the like based on the features extracted by the feature extraction layer. Specifically, an intermediate output vector from the trained model 420 (a vector output from the feature extraction layer and input to the fully connected layer next to the feature extraction layer) may be acquired as the feature vector F in the feature space. In other words, the feature vector F output from the trained model 420 is not limited to a vector that is ultimately output from the trained model 420 (final output), but may also be a vector that is intermediately output from the trained model 420 (intermediate output). Then, a hierarchical clustering process, a concept vector extraction process, or the like may be performed on the feature vector F. [Explanation of symbols]

[0329] 1. Image processing system 30 Image processing device (information processing device) 31 Controller 210,211,213,215 Person image (input image) 250,251,253,255,F feature vector 400,410,420 Learning Model 501 Separation plane 600,610,620 display screen C Concept G Cluster U Concept Vector

Claims

1. a control unit that acquires a plurality of feature vectors output from a machine-learned learning model corresponding to input of a plurality of input images to the learning model, generates a plurality of hierarchical clusters by executing a hierarchical clustering process on the plurality of feature vectors, and extracts a subspace or vector corresponding to a specific cluster from the plurality of clusters as a concept of the specific cluster; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the control unit extracts a representative vector relating to the specific cluster as a concept of the specific cluster.

3. 3. The information processing device according to claim 2, wherein the control unit generates a concept visualization image, which is a virtual input image corresponding to the representative vector and which visualizes a concept of the specific cluster, based on the representative vector and the learning model, and causes the display unit to display the concept visualization image.

4. a receiving unit that receives input of attribute information of the specific cluster; The information processing device according to claim 1 , further comprising:

5. The information processing device described in any one of claims 1 to 4, characterized in that when determining the similarity between the first image and the second image based on a first feature vector output from the learning model in response to an input of a first image to the learning model and a second feature vector output from the learning model in response to an input of a second image to the learning model, the control unit calculates the contribution of at least one concept among a plurality of concepts corresponding respectively to the plurality of clusters to the determination of the similarity between the first image and the second image.

6. The information processing device described in any one of claims 1 to 4, characterized in that when the control unit determines the similarity between the first image and the second image based on a first feature vector output from the learning model in response to a first image input to the learning model and a second feature vector output from the learning model in response to a second image input to the learning model, the control unit extracts, from among a plurality of subspaces corresponding respectively to the plurality of clusters, a subspace that relatively reduces the distance between the projection vectors of the first and second feature vectors onto the orthogonal complementary space, or a vector that spans the subspace, as a concept that serves as the basis for determining that the first image and the second image are not similar to each other.

7. a control unit that acquires a plurality of feature vectors output from a machine-learned learning model corresponding to input of a plurality of input images to the learning model, generates a plurality of hierarchical clusters by executing a hierarchical clustering process on the plurality of feature vectors, and determines two or more input images corresponding to a specific cluster among the plurality of clusters as a group of images representing the concept of the specific cluster; An information processing device comprising:

8. a) obtaining a plurality of feature vectors output from a machine-learned learning model in response to input of a plurality of input images to the learning model; b) generating a plurality of hierarchical clusters by performing a hierarchical clustering process on the plurality of feature vectors; c) extracting a subspace or vector corresponding to a specific cluster from among the plurality of clusters as a concept of the specific cluster; An information processing method comprising:

Images