Information processing apparatus, information processing method, and program

Through the classification and merging technology of information processing equipment, the problem of difficulty in removing noise elements in the image cluster is solved, the data accuracy is improved, and efficient extraction of data related to specified objects is achieved.

CN118901064APending Publication Date: 2024-11-05CANON KK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380028716.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-23
Filing Date
2023-03-03
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

The prior art is difficult to effectively reduce noise elements when extracting objects related to search keywords from image clusters, resulting in a large number of unrelated images contained in the data set.

Method used

Through information processing equipment, image elements are obtained and classified, similarity between elements is calculated, core clusters and merge clusters are selected, elements with high similarity are merged, and clusters are output after removing noise to improve data accuracy.

Benefits of technology

It realizes effective removal of noise elements in the data set related to the specified object, improves the extraction accuracy of signal elements, and reduces the impact of noise.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118901064A_ABST
    Figure CN118901064A_ABST
Patent Text Reader

Abstract

The present invention reduces noise included in a set of data related to a specified object by implementing a set of only object A data included in a cluster A by adding other clusters to a cluster having the highest reliability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to data collection technology. Background Art

[0002] Patent Document 1 discusses a technique for extracting images of objects related to a search keyword from a cluster of images including images of objects that are likely to be related to the keyword.

[0003] The present invention aims to reduce noise included in a collection of data related to a specified object.

[0004] Citation List

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent No. 6314071 Summary of the invention

[0007] Solutions for solving problems

[0008] In order to solve the above-mentioned problem, the information processing device according to the present invention includes: an acquisition unit, which is configured to acquire multiple elements; a classification unit, which is configured to classify the elements into multiple clusters based on the similarity between the acquired elements; a first selection unit, which is configured to select a first cluster related to a predetermined object based on a first indicator calculated from the elements included in the classified clusters; a second selection unit, which is configured to specify a second cluster based on a second indicator calculated from the first cluster and the elements included in the classified clusters; a designation unit, which is configured to specify a merged element to be merged in the acquired elements based on the second cluster and the first cluster; and an output unit, which is configured to output a third cluster, the third cluster including the merged element, the elements included in the first cluster, and the elements included in the second cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram showing an example of the hardware configuration of an information processing device.

[0010] Figure 2 is a block diagram showing a functional configuration example of an information processing device.

[0011] Figure 3 is a flowchart showing processing to be performed by the information processing device.

[0012] Figure 4 It is an activity diagram showing the processing to be performed by the information processing device.

[0013] Figure 5 is a schematic diagram showing an example of data and sequence information.

[0014] Figure 6 It is an activity diagram showing the processing to be performed by the information processing device.

[0015] Fig. 7A is a schematic diagram showing an example of the relationship between data.

[0016] Figure 7B is a schematic diagram showing an example of the relationship between data.

[0017] Figure 8 It is an activity diagram showing the processing to be performed by the information processing device.

[0018] Fig. 9 It is an activity diagram showing the processing to be performed by the information processing device.

[0019] Fig.10 It is an activity diagram showing the processing to be performed by the information processing device.

[0020] Fig.11 is a table showing examples of elements and indices.

[0021] Fig.12 It is an activity diagram showing the processing to be performed by the information processing device.

[0022] Fig.13 is a schematic diagram showing an example of the relationship between data.

[0023] Fig.14 It is an activity diagram showing the processing to be performed by the information processing device.

[0024] Fig.15 It is an activity diagram showing the processing to be performed by the information processing device.

[0025] Fig.16 2 is a schematic diagram showing an example of transformation processing for an image. DETAILED DESCRIPTION

[0026] In the existing hierarchical clustering which is one of the methods of grouping images of the same kind in clusters, an operation is performed so that a certain element (image) is included in the cluster with the highest probability. Therefore, for example, in the case where cluster A indicates object A and cluster B indicates object B, an element (image) similar to object B but indicating object A is included in cluster B. In addition, in the case of hierarchical clustering, due to duplication of integration with similar clusters, an element indicating object B may be included in the cluster indicating object A. In the present invention, in order to eliminate object B included in cluster A and realize a data set of only object A, other clusters are added to the cluster with the highest reliability.

[0027] Hereinafter, in the collected data, the elements to be extracted are referred to as signal elements, and the elements to be removed are referred to as noise elements. In the case of Patent Document 1, images of objects related to the keyword are signal elements, and other images are noise elements.

[0028] Hereinafter, preferred exemplary embodiments of the present invention are described with reference to the accompanying drawings.

[0029] <First Exemplary Embodiment>

[0030] Figure 1 is a diagram illustrating an example of a hardware configuration of an information processing apparatus according to the first exemplary embodiment. The information processing apparatus 100 collects data highly related to a designated query to generate a cluster for each object.

[0031] The central processing unit (CPU) 101 is a central processing unit that controls the entire information processing device 100. The read-only memory (ROM) 102 is a read-only memory that stores programs and parameters that do not need to be changed. The random access memory (RAM) 103 is a random access memory that temporarily stores programs and data supplied from an external device or the like. The external storage device 104 is a storage device such as a hard disk and a memory card that is fixedly installed on the information processing device 100. The external storage device 104 may include a flexible disk (FD) that can be attached to and detached from the information processing device 100, an optical disk such as a compact disk (CD), a magnetic card or an optical card, an integrated circuit (IC) card, and a memory card, etc. When the CPU 101 reads out the program stored in the ROM 102 and the external storage device 104 and executes the program, the functions and processing of the information processing device 100 described below are realized.

[0032] An input interface (I / F) 105 is an interface with an input unit 109 such as a pointing device and a keyboard to receive a user's operation and input data. An output device I / F 106 is an interface with a monitor (display device) 110 to display data held by the information processing apparatus 100 and supplied data. The data output method is not limited to a display device such as a monitor, and may be an output device such as a speaker that outputs audio. A system bus 108 is a transmission path that communicatively connects the units (101 to 107).

[0033] Figure 2 : is a block diagram illustrating a functional configuration example of the information processing apparatus 100 according to the present exemplary embodiment.

[0034] The information processing device 100 outputs the third cluster of the extracted signal elements, and outputs the result from the output unit 16. In order to actually implement the present invention, a means for using the output result is required. It is assumed that the result is output to, for example, a display device or an analysis device for face authentication, etc., but since the application is not limited, the description of the means for using the result is omitted.

[0035] The acquisition unit 11 acquires data to be processed, that is, data related to a specified object. More specifically, the acquisition unit 11 acquires a large number of facial images that are considered to be facial images of the same person. For example, in a person search system using a surveillance camera, facial images of people similar to each other obtained by specifying a query image and inputting a facial image of a person to be searched can be the data to be processed. Alternatively, as discussed in Patent Document 1, data extracted by searching using keywords such as names can be acquired from the Internet. The images (data) acquired by the acquisition unit 11 can also be referred to as elements in subsequent processing.

[0036] In the present typical embodiment, a facial image is acquired as an element, but the data to be handled is not limited to the facial image, and a collection of elements (data) considered to be of the same kind is handled. Other examples of elements considered to be of the same kind include: a photograph of a printed document or a handwritten document of a specified character (e.g., "A") and an image obtained by photographing a specified object (e.g., a car of a specified model). Images obtained by performing transformations such as enlargement / reduction / rotation and color correction on these images and cropping images of appropriate sizes from these images can be used as elements. As described above, the data collection method is not limited, and the acquired data has data including the specified object and other data (excluding the specified object) in a mixed manner.

[0037] In other words, elements are classified into signal elements and noise elements. More specifically, signal elements are elements to be extracted and are facial images of the same person. Noise elements are facial images of people different from the person of the signal element, etc. For example, in the above-mentioned person search system, most facial images (elements) obtained by inputting "facial image of Mr. A" are considered to be "facial images of Mr. A" (signal elements). Other images (i.e., "facial images of people other than Mr. A (similar to Mr. A)", facial illustrations, and images other than facial images) are noise elements.

[0038] The classification unit 12 classifies the acquired elements (images) into a plurality of clusters based on the similarity between the elements. In other words, the classification unit 12 clusters the set of predetermined elements acquired by the acquisition unit 11. Although the details are described below, not all elements are clustered, but only elements with strong relationships are extracted as one cluster or multiple clusters.

[0039] The first cluster selection unit 13 selects a core cluster (first cluster) related to the predetermined object from among the classified clusters based on a first index calculated from the elements included in the clusters. Hereinafter, a cluster that is likely to include only elements related to the predetermined object (i.e., signal elements) is referred to as a core cluster. Among the clusters classified by the classification unit 12, one cluster that is considered to include only signal elements is extracted as a core cluster. The details are described below.

[0040] The second cluster selection unit 14 specifies a merged cluster (second cluster) based on a second index calculated from elements included in the core cluster (first cluster) and elements included in a plurality of classified clusters. In other words, among the plurality of classified clusters, a cluster similar to the selected cluster is specified. The merged cluster is a cluster different from the core cluster and is a cluster to be merged with the core cluster. A cluster similar to the core cluster is specified as a merged cluster. Details are described below.

[0041] The merge element specifying unit 15 specifies the merge element to be merged among the acquired elements based on the merge cluster (second cluster) and the core cluster (first cluster). In other words, among the elements that are neither included in the core cluster nor in the merge cluster, the elements whose similarity with the core cluster or the merge cluster reaches a predetermined value (third threshold) or more are specified as merge elements. The details are described below.

[0042] The output unit 16 outputs a cluster (third cluster) including the merged elements, the elements included in the core cluster (first cluster), and the elements included in the merged cluster (second cluster). In other words, the elements of the core cluster, the elements of the merged cluster, and the merged elements are output as a list of elements after noise removal. The details are described below.

[0043] The storage unit 17 is the RAM 103 , and appropriately stores information necessary to perform the above-described processing of the acquisition unit 11 to the output unit 16 .

[0044] The following describes processing to be performed by the information processing apparatus 100. Figure 3 1 is a flowchart showing the processing to be performed by the information processing device 100. The processing corresponding to each flowchart described in the present exemplary embodiment can be implemented by software using a CPU, or can be implemented by hardware such as an electronic circuit, etc. In the following description, the letter "S" is added to the beginning of each processing (step) to omit the term "processing (step)".

[0045] First, in S300, the acquisition unit 11 acquires a plurality of data (elements). Figure 4 to describe the process.

[0046] In S300, the acquisition unit 11 acquires a plurality of data (elements). The acquired result is output as an element list O21. The element is the image itself, and is usually an image or an image file name loaded on a memory. In the case where information related to the element is required, an element related information list O22 is output. Examples of information related to the element include sequence information. The sequence information is the number 1, 2, ... assigned to the image in the order that most likely includes Mr. A when acquiring "Mr. A's facial image". Alternatively, the information related to the element may be certainty information. Certainty information is, for example, the degree of certainty as "Mr. A's facial image" represented by a numerical value between 0 and 1 (including both 0 and 1). Depending on the object for which the acquisition unit 11 acquires the element set, the sequence information and the certainty information are output. Although the information output is different depending on the object, for example, in the case of a person search system, if the degree of similarity with the facial image of the person to be searched is output, the degree of similarity is used as the certainty information. In addition, in the case of outputting facial images in descending order of similarity, the order of output is used as the sequence information.

[0047] Although the case where the element-related information list includes sequence information is described in the present typical embodiment, when the sequence information is obtained when only the certainty information can be obtained, for example, elements numbered from 1 in descending order of the certainty information can be used. The certainty information and the sequence information do not have to be unique, and multiple elements can have the same value. The element-related information is used in the processing of subsequent stages, but if the element-related information is not used, the element-related information list O22 may not be output. The graph elements beginning with the letter "O" are data to be handled in the input or output of each processing, and are stored in the storage unit 17.

[0048] Figure 5 An element related information list O22 is shown. Each circle in the figure indicates an element. The numbers given in the circles are sequence information from 1 to 9. Illustration of numbers above 10 is omitted. For the following description, E1 to E9 are assigned to some elements.

[0049] Return to reference Figure 3 In S301 , the classification unit 12 extracts elements having a strong relationship as clusters from the elements acquired by the acquisition unit 11 . Figure 6 The activity is shown in the diagram.

[0050] In S3010, the classification unit 12 extracts features (first feature quantities) from the data (elements) based on the first feature extraction method. The feature quantities are calculated for each element in the element list O21, and the feature quantities are output as the feature quantity list O41. The first feature extraction method may be an optional method. For example, a feature vector is obtained from a feature extraction trained model by using Deep Residual Learning for Image Recognition (ResNet) of a deep neural network.

[0051] In S3011, the classification unit 12 obtains similarity by comparing the features of the elements. More specifically, the similarity between two elements in the feature quantity list O41 is calculated in a round-robin manner, and a similarity list O42 is output. Here, the similarity is a value in the range from -1 to 1, but is not limited thereto. As a method for calculating the similarity, for example, the cosine similarity between the feature quantities can be used.

[0052] Fig. 7A is a schematic diagram showing the relationship between similarities of elements. Fig. 7A Shows elements represented by circles and line segments connecting each element. In other words, Fig. 7A is a diagram showing the contents of a similarity list O42 superimposed on an element list O21. In the diagram, two elements whose similarity is equal to or greater than a predetermined threshold value TH1 are connected by a solid line, and two elements whose similarity is a predetermined threshold value TH2 (>TH1) are connected by a bold line.

[0053] In S3012, the classification unit 12 extracts elements whose similarity is equal to or greater than a predetermined threshold to generate clusters. That is, clustering is performed based on the similarity list O42. In other words, two optional elements included in one cluster have a similarity equal to or greater than a threshold (TH2). The result of cluster classification is output as a cluster list O43.

[0054] Figure 7B is a schematic diagram showing the relationship between similarities of elements. Figure 7B 043 is a diagram showing a cluster list O43 superimposed on the element list O21 and the similarity list O42. In this example, three clusters of cluster A61, cluster B62, and cluster C63 are extracted.

[0055] Furthermore, the classification unit 12 also calculates the number of elements indicating the number of elements included in each cluster, and outputs the number of elements as an element number list 044. However, if the element number list 044 is not used in processing at a subsequent stage, the element number list 044 may not be output.

[0056] Return to reference Figure 3In S302, the first cluster selection unit 13 selects a core cluster (first cluster) based on information included in the elements included in each cluster. The cluster selected here is considered to include only the signal elements in the cluster list O43.

[0057] Figure 8 is a graph of the activity in the first cluster selection unit 13. The clusters in the cluster list O43 have a strong relationship and a high similarity equal to or greater than the threshold value TH2, and are expected to be a collection of elements indicating "obviously Mr. A" and "obviously Mr. A wearing a mask". A representative cluster of the element list O21 is to be selected from these clusters. In other words, a cluster indicating "obviously Mr. A" is to be selected. Since the element list O21 is a collection of elements of characters similar to Mr. A, it is considered that the cluster indicating "obviously Mr. A" can be selected by selecting based on the following criteria.

[0058] Guideline 11) Clusters that include elements with high similarity

[0059] Guideline 12) Clusters containing a large number of elements

[0060] Guideline 13) Clusters that include a large number of elements with smaller numbers in the order information (elements with high relevance)

[0061] Therefore, in S3020, the first cluster selection unit 13 calculates the first degree of certainty for each cluster based on the similarity between elements, the number of elements and the degree of association. An example of calculating the degree of certainty based on the cluster list, the similarity list, the element number list and the element related information list is described. In other words, the first degree of certainty (first index) is a degree of certainty indicating the possibility of including a predetermined object. Although not all policies must be used, clusters that meet more policies are more desirable. The first degree of certainty of each cluster in the cluster list O43 is calculated according to the above-mentioned policies 11 to 13. The first degree of certainty (first index) has, for example, a value in the range from 0 to 1, and a larger value indicates that the cluster is more "obviously Mr. A".

[0062] A method of calculating the first degree of certainty is described below.

[0063] When α1+β1+γ1=1, the following equation holds true:

[0064] [Equation 1]

[0065]

[0066] [Equation 2]

[0067]

[0068] [Equation 3]

[0069]

[0070] [Equation 4]

[0071] CC = α1·CC1 + β1·CC2 + γ1·CC3 (Equation 4)

[0072] Among them, S ele is the set of subscripts of the elements included in the cluster, S all is the set of subscripts of all elements, sim ij is the similarity between element i and element j, n(S) is the number of elements in set S, n(S ele ) is the number of elements in the cluster, n(S all ) is the number of all elements, n P k is the total number of permutations obtained by selecting k elements from n elements, and r i It is the sequence information of element i.

[0073] For example, the first degree of certainty CC1 in the case of guideline 11, the first degree of certainty CC2 in the case of guideline 12, and the first degree of certainty CC3 in the case of guideline 13 are respectively expressed by equations 1 to 3. The equations described here are illustrative, and the equations are not particularly limited as long as the equations express evaluation values ​​of guidelines 11 to 13.

[0074] The first degree of certainty CC of each cluster is calculated from the values ​​of CC1 to CC3. An example of calculating the first degree of certainty CC is represented by Equation 4. Note that α1, β1, and γ1 are predetermined values. In Equation 4, the values ​​of CC1 to CC3 based on guidelines 11 to 13 are used; however, indicators based on other guidelines may be added. The first degree of certainty CC of each cluster is output as a first degree of certainty list O71.

[0075] As an example, the above equation is used to determine the first degree of certainty CC of cluster A61, cluster B62, and cluster C63. To simplify the calculation, the similarity indicated by the thick line is set to 0.7, and the similarity indicated by the thin line is set to 0.3. In addition, to simplify the calculation, all search order information above 10 is treated as 10.

[0076] First, the first degree of certainty CC of cluster A61 is determined. The number of elements in cluster A61 is 9, the number of thick lines connecting the elements in cluster A61 is 16, and the number of thin lines is 1, from which CC1=0.319, CC2=0.360, and CC3=0.571 are calculated. In addition, when α1=0.3, β1=0.1, and γ1=0.6 are set, CC=0.474 (the first degree of certainty of cluster A61) is obtained.

[0077] Likewise, the first degree of certainty CC of the cluster B62 is obtained as CC=0.263 (first degree of certainty of the cluster B62). The first degree of certainty CC of the cluster C63 is obtained as CC=0.225 (first degree of certainty of the cluster C63).

[0078] In S3021, the first cluster selection unit 13 selects a cluster whose first degree of certainty is equal to or greater than the first threshold as a core cluster (first cluster) based on the first degree of certainty. Alternatively, the first cluster selection unit 13 selects a cluster with the largest first index as a core cluster. The first cluster selection unit 13 selects a cluster with the largest value of the first degree of certainty from the first degree of certainty list O71 as a core cluster O72. In the case where multiple clusters have the largest value of the first degree of certainty, one of the multiple clusters or the union of the multiple clusters is selected as the core cluster O72. In this example, since cluster A61 has the largest first degree of certainty (0.474), cluster A61 is selected.

[0079] When the maximum first certainty does not exceed the first threshold, the process may be terminated without selecting a core cluster. In this case, the process returns to the collection of elements or the extraction of features.

[0080] Return to reference Figure 3 In S303, the second cluster selection unit 14 specifies a cluster similar to the core cluster as a merged cluster (second cluster).

[0081] Fig. 9 The processing to be performed by the second cluster selection unit 14 and the data flow are shown.

[0082] For example, whether a cluster is a merged cluster may be specified based on the following criteria.

[0083] Guideline 21) A cluster including a large number of element pairs, wherein the element pairs are element pairs having a high similarity (similarity equal to or greater than a predetermined value) among element pairs of an element in a core cluster and an element included in the cluster.

[0084] Guideline 22) Clusters with high first certainty of clusters

[0085] The above-mentioned guidelines are examples for determining whether a cluster is similar to a core cluster, and other guidelines may be used. In other words, the merged cluster (second cluster) is selected by calculating the core cluster belonging degree (second index), which is an index indicating the similarity between the core cluster and each cluster other than the core cluster (first cluster). Since the second index of each cluster other than the core cluster is calculated, the cluster to be processed may also be referred to as an object cluster. In the following specific example, the object clusters are cluster B and cluster C.

[0086] In S3030, the second cluster selection unit 14 calculates the core cluster belonging degree for each cluster based on the elements included in each cluster and the elements included in the core cluster. The core cluster belonging degree of each cluster in the cluster list O43 is calculated according to the above-mentioned guidelines 21 and 22. The core cluster belonging degree has a value in the range from 0 to 1, for example, and a larger value indicates that the cluster is more "similar to the core cluster".

[0087] For example, the core cluster belonging degree corresponding to policy 21 is represented by CB1, and the core cluster belonging degree corresponding to policy 22 is represented by CB2. The following equations 5 to 8 represent examples of methods for calculating the core cluster belonging degrees. In the equations, α2, β2, and TH3 are predetermined values. Note that the equations are not particularly limited as long as the equations represent evaluation values ​​of policies 21 to 22.

[0088] [Equation 5]

[0089] N sim =n({sim ij |sim ij ≥TH1, i∈Sele, j∈Score, i<j})

[0090] α2+β2=1 (Equation 5)

[0091] [Equation 6]

[0092]

[0093] [Equation 7]

[0094] CB2=CC (Equation 7)

[0095] [Equation 8]

[0096] CB = α2·CB1 + β2·CB2 (Equation 8)

[0097] Among them, S ele is the set of subscripts of the elements included in the cluster, S core is the set of subscripts of all elements, sim ij is the similarity between element i and element j, n(S) is the number of elements in set S, n(S ele ) is the number of elements in the cluster, n(S core ) is the number of all elements, N sim is the number of elements having a similarity equal to or greater than TH1 in the similarities between the elements of the cluster and the elements of the core cluster, U(x) is a step function (0 when x<0, 1 when x≥0), and CC is the core cluster certainty of the cluster.

[0098] The core cluster belonging degree CB (second index) of each cluster is calculated from the values ​​of CB1 and CB2. When the core cluster belonging degree CB relative to the core cluster A is calculated for cluster B and cluster C, CB=0.160 is calculated for cluster B, and CB=0.023 is calculated for cluster C. In equation 8, the values ​​of CB1 and CB2 based on guidelines 21 and 22 are used; however, indicators based on other guidelines can be added. The core cluster belonging degree CB of each cluster is output as a core cluster belonging degree list O101.

[0099] Refer to the above equation and Fig. 7A and Figure 7B , the processing in S3030 is described in more detail. First, the core cluster belonging degree CB of cluster B62 is determined. Nsim represents Figure 7B The number of solid lines connecting the elements in cluster A 61 and the elements in cluster B 62 is shown. Figure 7B It can be read as Nsim=4. When Nsim is small, the relationship between clusters is not necessarily large, so the core cluster belonging degree CB1 is set to zero. This is represented by the unit step function in Equation 5. When TH3=3 is set, Nsim≥TH3 holds, and CB1=0.148 is obtained from Equation 6.

[0100] The core cluster belonging degree CB2 is equal to the first certainty degree of cluster B62 and is 0.263. When α2=0.9 and β2=0.1 are set, 0.160 is obtained as the core cluster belonging degree CB of cluster B62. In other words, from Equation 8, CB=0.160 (core cluster belonging degree of cluster B62) is calculated.

[0101] Subsequently, the core cluster degree CB of cluster C63 is determined. There is no solid line connecting the elements in cluster A61 and the elements in cluster C63, so Nsim=0 is calculated from equation 5. Since the value of the unit step function in equation 6 is zero, CB1=0 is calculated from equation 6. The core cluster degree CB2 is equal to the first determination degree of cluster C63, and according to equation 7, the core cluster degree CB2 is 0.225. Based on the foregoing, when equation 8 is used to determine the core cluster degree CB of cluster C63, 0.023 is calculated. In other words, the core cluster degree of cluster C relative to core cluster A is calculated to be CB=0.023 (core cluster degree of cluster C63). The second cluster selection unit 14 puts the core cluster degrees of each cluster into a list together, and outputs the core cluster degree list O101. Since cluster A is a core cluster, the calculation of the core cluster degree can be omitted.

[0102] exist Fig. 9In S3031 shown, the second cluster selection unit 14 selects a cluster whose core cluster belonging degree is greater than a second threshold as a merged cluster (second cluster) based on the core cluster belonging degree (second index) calculated for each cluster in the plurality of clusters. In the core cluster belonging degree list O101, a cluster whose core cluster belonging degree is equal to or greater than the second threshold TH4 is designated as a merged cluster.

[0103] When the threshold value TH4 is set to 0.1, only cluster B62 is designated as the merged cluster from the above calculation results. Generally, multiple clusters can be designated as merged clusters; however, depending on the value of the core cluster belonging degree, only one cluster is designated or no cluster is designated in some cases. The clusters designated as merged clusters are output as the merged cluster list O102.

[0104] Return to reference Figure 3 Flowchart shown. In S304, the merge element designation unit 15 designates an element similar to the core cluster among the elements that are neither included in the merged cluster nor in the core cluster as a merged element. The merge cluster designation unit 14 determines the similarity between each cluster and the core cluster as an indicator (second indicator) indicating the degree of belonging to the core cluster, and designates the merged cluster from the value of the indicator. The merge element designation unit 15 determines the similarity between each element and the cluster of interest (core cluster or merged cluster) as an indicator (third indicator) indicating the degree of belonging to the cluster, and designates the merged element from the value of the indicator. In this example, an element whose degree of belonging (third indicator) relative to the core cluster or the merged cluster is greater than the third threshold is used as a merged element. An element whose statistical value (for example, average value) of the third indicator is equal to or greater than a reference value can be determined as a merged element without using a threshold.

[0105] Fig.10 1 shows the processing and data flow to be performed by the merge element specifying unit 15. For example, whether an element is a merge element can be specified based on the following criteria.

[0106] Guideline 31) Elements that have a large number (predetermined number or more) of element pairs with high similarity to elements in the cluster of interest

[0107] The above guidelines are examples for determining whether an element is similar to a cluster of interest, and other guidelines may be used.

[0108] In S3040, the merge element specifying unit 15 determines the cluster of interest from the core cluster and the merged cluster. The cluster of interest is extracted from the core cluster O72 and the merged cluster list O102. Since the core cluster is cluster A61 and the merged cluster is cluster B62, these two clusters are sent to subsequent processing. The processing in S3041 is performed on each cluster of interest. First, cluster A61 is processed.

[0109] In S3041, the merged element specifying unit 15 calculates the cluster belongingness of each element that is neither included in the merged cluster nor in the core cluster. The cluster belongingness has a value, for example, in the range from 0 to 1, and a larger value indicates that the element is more "similar to the cluster of interest". For example, the cluster belongingness CBE1 corresponding to the guideline 31 is represented by the following equations 9 to 11. In the equations, α3 and TH5 are predetermined values. Note that the described equations are illustrative, and the equations are not particularly limited as long as the equations represent the evaluation values ​​of the guideline 31.

[0110] [Equation 9]

[0111] N esim =n({sim ij |sim ij ≥TH1, i=I ele , j∈S tgt ,i <j})

[0112] α3=1 (Equation 9)

[0113] [Equation 10]

[0114]

[0115] [Equation 11]

[0116] CBE=α3·CBE1 (Equation 11)

[0117] Among them, I ele is the subscript of the element, S tgt is the set of subscripts of the elements included in the object cluster, sim ij is the similarity between element i and element j, n(S) is the number of elements in set S, n(S tgt ) is the number of elements in the object cluster, N esim is the number of similarities equal to or greater than TH1 among the similarities of the elements in the object cluster, and U(x) is a step function (0 when x<0, 1 when x≥0).

[0118] The cluster degree CBE is calculated from the value of CBE1. In Equation 11, only the value of CBE1 based on the guideline 31 is used; however, indicators based on other guidelines may be added. The cluster degree CBE of each element with respect to the cluster of interest is output as a cluster degree list O131.

[0119] Reference Figure 7B Describes the state of the specified merged element. First, determine the cluster belonging degree CBE of element E1 relative to cluster A61. Nesim represents Figure 7B The number of solid lines connecting element E1 to the elements in cluster A61 is shown. Figure 7B It can be read as Nesim=3. In addition, the number of elements in cluster A61 is 9. When Nesim is small, the relationship between element E1 and cluster A61 is not large, so the cluster belonging degree CBE1 is determined to be zero. This is represented by the unit step function in equation 9. When TH5=3 is set, Nesim≥TH5 holds, so CBE1=0.333 is obtained from equation 10. Then, 0.333 is obtained from equation 11 as the cluster belonging degree CBE. In a manner similar to CBE=0.333 (the cluster belonging degree of element E1 to cluster A61), the cluster belonging degrees of each element E2 to E9 relative to cluster A61 are determined, and the cluster belonging degrees are output as the cluster belonging degree list O131 (see Fig.11 After processing cluster A, cluster B is processed similarly, and a cluster belongingness list O31 is output (see Fig.11 ).

[0120] Next, in S3042, the merged element is designated from the result of the cluster belonging degree list O131. If the cluster belonging degree of the element relative to any cluster of interest is equal to or greater than the predetermined threshold TH6, the element is designated as the merged element. There may be multiple clusters of interest that meet the conditions. When the threshold TH6 is set to 0.3, the four elements E1, E2, E3 and E4 are designated as merged elements (see Fig.11 ). The four elements designated as merge elements are output as a merge element list O132.

[0121] Return to reference Figure 3 In S305, the output unit 16 outputs the cluster after noise removal. In other words, it outputs a list of elements after noise removal. This is a list of elements that are considered to be signal elements. Fig.12 The flow of the processing to be performed by the output unit 16 is shown. In S305, the output unit 16 collects the elements of the core cluster O72, the elements of the merged cluster, and the merged elements, and outputs these elements as the noise-removed element list O171 (third cluster). The elements of the merged cluster can be the elements included in the merged cluster extracted from the merged cluster list O102.

[0122] Reference Figure 7B Describe the operation.

[0123] The elements of the core cluster are the elements included in the core cluster. Figure 7B In the example, the core cluster is cluster A61. The nine elements included in cluster A61 are extracted.

[0124] The elements of a merged cluster are the elements included in the merged cluster. Figure 7B The merged cluster is only cluster B62. The three elements included in cluster B62 are extracted.

[0125] The merged element is an element that is included in neither the merged cluster nor the core cluster and has been specified to be merged into the core cluster based on the third index. Figure 7B In , the merged elements are 4 elements E1, E2, E3 and E4. Extract these 4 elements.

[0126] The extracted elements (16 elements) are collected into the noise-removed element list O171, and the noise-removed element list O171 is output. Fig.13 The result is shown. The elements included in the noise-removed element list O171 are represented as solid black circles. In the above manner, the noise elements can be removed from the element set in which the noise elements and the signal elements are mixed, and the signal elements can be extracted with high accuracy.

[0127] Providing the second cluster selection unit has the effect of extracting signal elements that are not extracted by the prior art. In addition, clusters of elements that are strongly similar to each other are created by the classification unit, which has the effect of reducing noise elements extracted by the prior art. As a result, the noise included in the set of data related to the specified object can be reduced.

[0128] <Second Exemplary Embodiment>

[0129] In the first exemplary embodiment, the similarity used for clustering is used to calculate the core cluster belonging degree (second index). However, for example, in the case where it is expected that cluster A61 indicates "clearly Mr. A" and cluster B62 indicates "clearly Mr. A wearing a mask" as a result of clustering, it is more convenient to use the similarity calculated from the feature amount in the state of wearing a mask. In other words, instead of the feature amount used for clustering, it is desirable to calculate the similarity using the feature amount of the face wearing a mask (i.e., a feature amount that is insensitive to the texture of the mouth covered by the mask).

[0130] In other words, as a method of calculating the features of the elements, it is expected that two different calculation methods are used to obtain the feature quantities. The second cluster selection unit 14 obtains the core cluster belonging degree (second index) of each element based on the feature quantity obtained by the second method different from the first method. For example, in a case where the first method is a process for extracting features related to the face from the entire facial image, the second method may be a process for extracting features from an image obtained by performing a predetermined transformation process on a predetermined facial area. In other words, by changing a part of the process in the second cluster selection unit 14, a process for calculating the similarity from a feature quantity different from the feature quantity used for clustering and calculating the core cluster belonging degree may be performed. Refer to Fig.14 Give instructions. Fig.14 and Fig. 9The difference is that the similarity list O42 is changed to a second similarity list O192, and a process for creating the second similarity list O192 is added. Therefore, only this change will be described.

[0131] In S3032 , the second cluster selecting unit 14 calculates the second feature quantity for each element, and outputs the second feature quantity list O191 .

[0132] exist Figure 6 In S3010 shown, the feature quantities of all elements are calculated, while in S3032, instead of calculating the second feature quantities of all elements, it is sufficient to calculate the second feature quantities of only the elements included in the cluster. In other words, the second feature quantities are calculated for each element belonging to the cluster included in the cluster list O43. The elements included in the cluster are acquired from the cluster list O43.

[0133] In S3033 , the second cluster selection unit 14 calculates the similarity between the second feature quantities by using the second feature quantity list O191 , and outputs a second similarity list O192 .

[0134] In the above manner, when a specific trend is expected in the clusters, signal elements can be extracted with high accuracy by specifying a merged cluster using a feature amount consistent with the trend. In other words, noise included in a collection of data related to a specified object can be reduced.

[0135] <Third exemplary embodiment>

[0136] In a second typical embodiment, the core cluster belonging degree is calculated based on feature quantities obtained by multiple feature extraction methods. At this time, a feature quantity that is insensitive to the texture of the mouth covered by the mask is calculated. In a third typical embodiment, a feature extraction processing method for obtaining a second feature quantity different from the first feature in a simple method without preparing different feature extraction methods is described. In the case of using deep learning to train a feature quantity extractor, a large amount of time is generally required, and the amount of memory occupied by the feature quantity extractor itself is doubled during feature extraction. Therefore, an image in which the portion including the mouth is covered in the facial image of each element is synthesized, and the feature quantity of the synthesized image is calculated by the same processing as that used in S3030.

[0137] In other words, the second cluster selection unit 14 obtains the second feature amount different from the first feature amount by performing predetermined processing on each element. Fig.15A variation of the process in S3032 to be performed by the second cluster selection unit 14 is shown. In S30320, the second cluster selection unit 14 extracts the elements (facial images) included in each cluster from the cluster list O43. The feature quantity of each extracted element (facial image) is calculated through the processes in S30321 and S30322, and the result is output to the second feature quantity list O191. In S30321, the second cluster selection unit 14 performs a predetermined transformation process on the element (facial image) (O201). In this process, the facial image in which the part (specified area) including the mouth is covered is synthesized with the facial image (O202). In S30322, the second cluster selection unit 14 calculates the feature quantity of the processed facial image O202 by using the same feature extraction model as the first feature extraction method. Fig.16 An example of an element (facial image) O201 and an element (processed facial image) O202 is shown. In the present exemplary embodiment, as a specific example of the designated area, the mouth is covered in S30321; however, the designated area is not limited thereto. In the case where an image with sunglasses is expected to be obtained as a cluster, the portion including the eyes may be covered. In addition, an image obtained by superimposing and synthesizing a mask or sunglasses may be used instead of an image in which the portion including the mouth or eyes is covered. Fig.16 These examples are shown. As a predetermined transformation process, a process for covering a designated area of ​​an element is described. The predetermined transformation process is not limited to covering, superimposing, and synthesizing a portion of an image including a mouth or an eye. For example, a portion including an earlobe may be covered, or an ear ornament may be superimposed and synthesized. Alternatively, a process for blurring a designated area may be performed.

[0138] In the above manner, even when a specific trend is expected in a cluster, a signal element can be extracted with high accuracy. In addition, since one feature quantity extractor is used, it is possible to prevent a lot of time from being spent on creating the feature quantity extractor and to prevent an increase in the amount of memory required for execution.

[0139] <Other Typical Embodiments>

[0140] In all of the above-described typical embodiments, the case related to an image is described; however, the effect of the present patent is not limited to an image, and therefore, the scope of the present patent is not limited to an image.

[0141] The present invention is also realized by performing the following processing. That is, the software (program) for realizing the functions of the above-mentioned exemplary embodiments is provided to the system or device through a network for data communication or various types of storage media. In addition, the computer (or CPU, microprocessing unit (MPU), etc.) of the system or device reads out and executes the program. The program can be provided by being recorded in a computer-readable recording medium.

[0142] The present invention is not limited to the above-described exemplary embodiments, and the present invention may be changed and modified in various ways without departing from the spirit and scope of the present invention. Therefore, in order to disclose the scope of the present invention, the following claims are attached.

[0143] This application claims the benefit of Japanese Patent Application No. 2022-047586, filed on Mar. 23, 2022, which is hereby incorporated by reference herein in its entirety.

Claims

1. An information processing device, include: an acquisition unit configured to acquire a plurality of elements; a classification unit configured to classify the elements into a plurality of clusters based on the acquired similarities between the elements; a first selection unit configured to select a first cluster related to a predetermined object based on a first index calculated from elements included in the classified clusters; a second selection unit configured to specify a second cluster based on a second index calculated from elements included in the first cluster and the classified clusters; a specifying unit configured to specify a merging element to be merged among the acquired elements based on the second cluster and the first cluster; as well as An output unit is configured to output a third cluster, wherein the third cluster includes the merged element, the element included in the first cluster, and the element included in the second cluster.

2. The information processing device according to claim 1, in, The specifying unit calculates a third index of an element among the acquired elements that is included in neither the first cluster nor the second cluster, and The element having the third index equal to or greater than a third threshold is designated as the merged element.

3. The information processing device according to claim 2, in, The third index is an index calculated based on similarities with the elements included in the first cluster and the elements included in the second cluster.

4. The information processing device according to any one of claims 1 to 3, in, The classification unit generates a cluster by extracting elements whose similarity to each other is equal to or greater than a predetermined threshold.

5. The information processing device according to any one of claims 1 to 4, in, The first index is a certainty indicating a possibility that a predetermined object is included, and is calculated based on at least one of a similarity between elements, the number of elements included in a cluster, and information related to the elements.

6. The information processing device according to any one of claims 1 to 5, in, The first selection unit selects a cluster having the first index greater than a first threshold as the first cluster based on the first index calculated for each of the plurality of clusters.

7. The information processing device according to any one of claims 1 to 5, in, The first selection unit selects a cluster having the largest first index as the first cluster based on the first index calculated for each of the plurality of clusters.

8. The information processing device according to any one of claims 1 to 7, in, The second selection unit selects a cluster having the second index greater than a second threshold as the second cluster based on the second index calculated for each of the plurality of clusters.

9. The information processing device according to any one of claims 1 to 8, in, The second index is an index indicating a similarity between the first cluster and an object cluster different from the first cluster, and is calculated for each cluster.

10. The information processing device according to claim 9, in, The second index is calculated based on a similarity between an element included in the first cluster and an element included in the object cluster different from the first cluster, or the first index of the object cluster.

11. The information processing device according to any one of claims 1 to 10, in, calculating the first index based on a first feature amount extracted for each of the elements included in the plurality of clusters using a first method, and The second index is calculated based on a second feature amount extracted for each of the elements included in the plurality of clusters according to a second method different from the first method.

12. The information processing device according to claim 11, in, The elements are each a facial image, wherein the first method is a process for extracting features from the facial image, and Among them, the second method is a process for extracting features from an image obtained by performing a predetermined transformation process on a designated area of ​​the facial image.

13. An information processing method, include: The acquisition step is used to obtain multiple elements; A classification step for classifying the elements into a plurality of clusters based on the similarities between the acquired elements; a first selection step for selecting a first cluster related to a predetermined object based on a first index calculated from elements included in the classified clusters; a second selection step for specifying a second cluster based on a second index calculated from elements included in the first cluster and the classified clusters; A specifying step, for specifying a merge element to be merged from the acquired elements based on the second cluster and the first cluster; as well as The output step is used to output a third cluster, wherein the third cluster includes the merged element, the elements included in the first cluster, and the elements included in the second cluster.

14. A program for causing a computer to function as each unit included in the information processing apparatus according to any one of claims 1 to 12.