Processing apparatus, processing method, and program

Abstract

The present invention provides a processing device comprising: an acquisition unit that acquires a search query; a search unit that matches the search query with a plurality of sample images and searches for a sample image that matches the search query; a registration unit that registers the matching results in a matching history indicating one or a plurality of sample images matched in each matching; and an aggregation output unit that generates and outputs aggregation information regarding the matching results for a selected sample image among the plurality of sample images on the basis of the matching history.

Description

Technical Field

[0001] The present invention relates to a processing apparatus, a processing method, and a program. Background Art

[0002] Technologies related to the present invention are disclosed in Patent Documents 1 and 2. The technologies disclosed in Patent Documents 1 and 2 create a database storing images representing persons in various postures. Then, the technology searches the database for images matching a search query specified by a user and outputs the searched images. Prior Art Documents Patent Documents

[0003] Patent Document 1 International Publication No. 2021 / 084677 Patent Document 2 Japanese Unexamined Patent Publication No. 2019-091138 Summary of the Invention Problem to be Solved by the Invention

[0004] In a technology for searching a database for images matching a search query and outputting the images, what kind of images are registered in the database is an important factor.

[0005] Registering a large number of various images in the database increases the probability of searching for an image that matches a search query. For example, registering images of persons in various postures (e.g., sitting posture, standing posture, etc.) in the database increases the probability of searching for an image that matches a search query. Furthermore, instead of registering only one image of a certain type (e.g., an image of a sitting posture) in the database, registering a large number of images with multiple variations increases the probability of searching for an image that matches a search query.

[0006] On the other hand, the more images registered in the database, the greater the storage capacity required to store those images. As a result, the burden of costs and the effort required for data management increases.

[0007] Taking these points into consideration, it is necessary to optimize the image set to be registered in the database. Optimization involves tasks such as deleting unnecessary images from the database or adding missing images.

[0008] However, a certain number of images are registered in the database. Therefore, it is not easy to grasp the overall picture of the registered image collection and identify which images are superfluous and which are missing. Thus, there is a need for technology that can support the optimization of the image collection registered in the database.

[0009] The technologies disclosed in Patent Documents 1 and 2 are both technologies for searching a database for images that match a search query and outputting them. However, neither Patent Documents 1 nor 2 can assist in optimizing the image set registered in the database.

[0010] One example of the object of the present invention is, in view of the above-mentioned problems, to provide a processing device, a processing method, and a program that solve the problem of supporting the optimization of the image group registered in a database in a technology that searches for and outputs images that match a search query from a database. [Means for solving the problem]

[0011] According to one aspect of the present invention, Methods for obtaining search queries, A search means that compares the search query with a plurality of sample images and searches for the sample image that matches the search query, A registration means for registering the results of the matching in a matching history that shows one or more of the sample images that were matched in each of the aforementioned matchings, Summary output means that generates and outputs summary information regarding the results of the comparison for a selected sample image from a plurality of sample images, based on the comparison history. A processing device having the following is provided.

[0012] According to one aspect of the present invention, One or more computers, Get the search query, The search query is compared with multiple sample images, and the sample images that match the search query are searched for. The results of the matching are registered in the matching history, which shows one or more of the sample images that were matched in each of the aforementioned matching attempts. A processing method is provided for generating and outputting aggregated information regarding the matching results for a selected sample image from among multiple sample images, based on the matching history.

[0013] According to one aspect of the present invention, Computers, A means of obtaining search queries. A search means that compares the search query with a plurality of sample images and searches for the sample image that matches the search query. A registration means for registering the results of the matching in a matching history that shows one or more of the sample images that were matched in each of the aforementioned matchings, Aggregation output means that generates and outputs aggregated information regarding the matching results for a selected sample image from a plurality of sample images based on the matching history, A program is provided to enable it to function as such. [Effects of the Invention]

[0014] According to one aspect of the present invention, a processing device, a processing method, and a program are realized that solve the problem of supporting the optimization of the image group registered in a database in a technology that searches for and outputs images that match a search query from a database. [Brief explanation of the drawing]

[0015] The above object and other objects, features, and advantages will be described below Suitable embodiments and the accompanying drawings described below.

[0016] [Figure 1] It is a diagram illustrating an example of a functional block diagram of a processing device. [Figure 2] It is a diagram illustrating an example of the hardware configuration of a processing device. [Figure 3] It is a diagram schematically illustrating an example of a matching history generated by a processing device. [Figure 4] It is a diagram schematically illustrating an example of aggregate information generated by a processing device. [Figure 5] It is a flowchart illustrating an example of a processing flow of a processing device. [Figure 6] It is a flowchart illustrating another example of a processing flow of a processing device. [Figure 7] It is a diagram schematically illustrating another example of a matching history generated by a processing device. [Figure 8] It is a diagram schematically illustrating an example of information output by a processing device. [Figure 9] It is a diagram schematically illustrating another example of information output by a processing device. [Figure 10] It is a diagram schematically illustrating another example of information output by a processing device. [Figure 11] It is a diagram schematically illustrating another example of information output by a processing device. [Figure 12] It is a diagram schematically illustrating another example of information output by a processing device. DETAILED DESCRIPTION OF EMBODIMENTS

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all drawings, the same reference numerals are assigned to similar constituent elements, and the description thereof will be omitted as appropriate.

[0018] <First Embodiment> Figure 1 is a functional block diagram showing an overview of the processing device 10 according to the first embodiment. The processing device 10 includes an acquisition unit 11, a search unit 12, a registration unit 13, and an aggregation output unit 14.

[0019] The acquisition unit 11 acquires the search query. The search unit 12 compares the search query with multiple sample images and searches for a sample image that matches the search query. The registration unit 13 registers the matching results in the matching history, which shows one or more sample images that matched in each matching. The aggregation output unit 14 generates and outputs aggregated information regarding the matching results for the selected sample image from the multiple sample images, based on the matching history.

[0020] In this way, the processing unit 10 performs a search process to find sample images that match the search query from the database. Then, the processing unit 10 registers the results of the matching between the search query and the sample images in each search process as a matching history. 、 Using this matching history, aggregated information regarding the matching results is generated and output for each sample image. This aggregated information includes, for example, the number of times a match was achieved in past search processes. Based on this aggregated information, users can identify which images are redundant and which images are missing.

[0021] Thus, the processing device 10 of this embodiment solves the problem of supporting the optimization of the image group registered in the database in a technology that searches for and outputs images that match a search query from the database.

[0022] <Second Embodiment> "overview" The processing unit 10 of this embodiment is an embodiment of the processing unit 10 of the first embodiment. In this embodiment, the configuration of the processing unit 10 is made concrete. In addition, in this embodiment, the "aggregated information" described in the first embodiment is made concrete.

[0023] "Hardware configuration" Next, an example of the hardware configuration of the processing unit 10 will be described. Each functional unit of the processing unit 10 is realized by any combination of hardware and software. It will be understood by those skilled in the art that there are various modifications to the implementation method and the device. The software includes programs that are pre-installed at the time of shipment of the device, as well as programs downloaded from recording media such as CDs (Compact Discs) or from servers on the Internet.

[0024] Figure 2 is a block diagram illustrating the hardware configuration of the processing unit 10. As shown in Figure 2, the processing unit 10 includes a processor 1A, memory 2A, input / output interface 3A, peripheral circuitry 4A, and bus 5A. Peripheral circuitry 4A includes various modules. The processing unit 10 does not necessarily have peripheral circuitry 4A. The processing unit 10 may also be composed of multiple physically and / or logically separated devices. In this case, each of the multiple devices may have the above hardware configuration.

[0025] Bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuits 4A, and input / output interface 3A to send and receive data to and from each other. Processor 1A is a processing unit such as a CPU or GPU (Graphics Processing Unit). Memory 2A is a memory such as RAM (Random Access Memory) or ROM (Read Only Memory). Input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. Input / output interface 3A also includes interfaces for connecting to communication networks such as the Internet. Input devices include, for example, keyboards, mice, microphones, physical buttons, and touch panels. Output devices include, for example, displays, speakers, printers, and mailers. Processor 1A can issue commands to each module and perform calculations based on their calculation results.

[0026] "Functional Configuration" Next, the functional configuration of the processing device 10 of this embodiment will be described in detail. Figure 1 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 has an acquisition unit 11, a search unit 12, a registration unit 13, and an aggregation output unit 14.

[0027] The acquisition unit 11 acquires a search query. The search query is, for example, a query image. The query image may be a still image or a moving image. In addition, the search query may be data that describes the features of the query image in text or other format. The features of the query image are the features of the objects depicted in the query image. The objects depicted in the query image may be, but are not limited to, people, other animals, buildings, natural objects, artificial objects, etc. The features of the objects depicted in the query image may be, but are not limited to, facial features, body features, clothing features, belongings, posture features, gait features, movement features, movement trajectory, size, position, color, shape, feature point information, etc.

[0028] Here, an example of how the acquisition unit 11 can acquire a search query is described. For example, a user may input a search query through the input device of the processing unit 10. The acquisition unit 11 may then acquire the input search query. Alternatively, the processing unit 10 may be a server. The acquisition unit 11 may then acquire a search query sent from a client terminal. Alternatively, images generated by a surveillance camera may be input to the processing unit 10. The acquisition unit 11 may then acquire the images generated by the surveillance camera as a search query. Note that the means by which the acquisition unit 11 can acquire a search query are not limited to the examples shown here.

[0029] The search unit 12 searches for sample images that match the search query acquired by the acquisition unit 11.

[0030] Multiple sample images corresponding to the search objective are pre-stored in the storage device. For example, if the search objective is "to search for images of a person in the posture indicated by the search query," sample images of people in various postures are stored in the storage device. Also, if the search objective is "to identify the location where an image was taken," sample images taken at various locations are stored in the storage device. The processing unit 10 may also be equipped with the above-mentioned storage device. In addition, an external device configured to communicate with the processing unit 10 may also be equipped with the above-mentioned storage device.

[0031] The search unit 12 compares the search query acquired by the acquisition unit 11 with each of the multiple sample images and calculates the similarity. Then, based on the calculated similarity, the search unit 12 identifies a sample image that matches the search query. The condition for matching with the search query is that "the similarity is equal to or greater than a certain threshold value." The matching and calculation of similarity can be achieved using any conventional technology. For example, the technology described in Patent Document 1 can be used.

[0032] The registration unit 13 registers the results of the matching performed by the search unit 12 in the matching history, which shows one or more sample images that were matched in each matching.

[0033] Figure 3 schematically shows an example of a matching history. The matching history shown links the matching number with the matched sample image.

[0034] The "matching number" is information that identifies multiple matching instances that have been performed to date.

[0035] The "Matched Sample Images" column stores the identification information of the sample images that matched the search query in each matching process.

[0036] As mentioned above, the condition for matching a search query is that "the similarity is above a certain threshold." Therefore, as shown in Figure 3, multiple sample images may match simultaneously in each matching process.

[0037] Returning to Figure 1, the aggregation output unit 14 generates and outputs aggregated information regarding the matching results for the selected sample image among multiple sample images, based on the matching history.

[0038] The selected sample image is at least one sample image selected from among multiple sample images. The user may make this selection. Alternatively, the aggregation output unit 14 may make this selection according to any predetermined rules. An example of a user making this selection will be described in detail in the following embodiment.

[0039] The aggregated information regarding the selected sample image will show at least one of the following pieces of information: • Number of matches Matching rate • Statistics on the number of other sample images that were matched simultaneously. • Other sample images that have been matched at the same time • Number of times each of the other sample images that have been matched simultaneously • Sample images that have been matched simultaneously with other sample images and whose number of simultaneous matches meets the specified conditions. • Similarity with other sample images that have been matched at the same time.

[0040] The "number of matches" is the number of times the search query matched in the multiple matches shown in the matching history in Figure 3. The number of matches can be calculated, for example, by counting the number of times the identification information of the selected sample image is registered in the "Matched Sample Images" column of the matching history shown in Figure 3.

[0041] The "matching rate" is calculated by dividing the number of matches mentioned above by the total number of matches shown in the matching history in Figure 3.

[0042] The "statistical value of the number of other sample images that matched simultaneously" is the statistical value of the number of other sample images that matched at the same time as the selected sample image. The statistical value can be the mean, maximum, minimum, median, mode, etc. As mentioned above, the condition for matching with the search query is that "the similarity is equal to or greater than the threshold value." Therefore, as shown in Figure 3, multiple sample images may match simultaneously in each matching.

[0043] "Other sample images that have been matched at the same time" refers to sample images that have been matched at the same time as the selected sample image.

[0044] "Number of times each of the other sample images that have been matched simultaneously" indicates the number of times each sample image that has been matched simultaneously with the selected sample image has been matched simultaneously with the selected sample image.

[0045] "A sample image that has been matched simultaneously with other sample images and whose number of simultaneous matches meets a predetermined condition" refers to a sample image whose number of simultaneous matches with the selected sample image meets a predetermined condition. The predetermined condition is, for example, a predetermined number or more. Alternatively, other sample images may be ranked in descending order of the number of simultaneous matches with the selected sample image. The predetermined condition may then be that the sample image is within the top predetermined percentage, the top predetermined number, or the standard rank within that rank.

[0046] The "similarity to other sample images that have been matched simultaneously" is the similarity between the selected sample image and each of the other sample images that have been matched simultaneously with it. The calculation of the similarity between the two images is achieved using all conventional techniques.

[0047] The aggregation output unit 14 generates aggregated information about the selected sample images as described above, based on the matching history shown in Figure 3. The aggregation output unit 14 then outputs the generated aggregated information about the selected sample images. For example, the aggregation output unit 14 may output the aggregated information via an output device such as a display, projection device, or printer. Alternatively, the processing unit 10 may be a server. The aggregation output unit 14 may also transmit the aggregated information to a client terminal. In this case, the output device of the client terminal outputs the aggregated information.

[0048] The aggregation output unit 14 may generate and output aggregation information for the selected sample image each time a selected sample image is selected from among multiple sample images. Alternatively, the aggregation output unit 14 may pre-generate aggregation information for each of the multiple sample images based on the latest matching history, as shown in Figure 4. Then, when a selected sample image is selected from among multiple sample images, the aggregation output unit 14 may extract the aggregation information for the selected sample image from the aggregation information for each of the multiple sample images and output it.

[0049] Next, an example of the processing flow of the processing device 10 will be explained using the flowcharts in Figures 5 and 6.

[0050] First, using the flowchart in Figure 5, we will explain an example of the process flow for generating a matching history as shown in Figure 3.

[0051] When the processing unit 10 receives a search query (S10), it compares the search query with multiple sample images and searches for a sample image that matches the search query (S11). Next, the processing unit 10 outputs the search results showing the matched sample image (S12). The processing unit 10 also registers the result of the comparison in S11 in the comparison history, which shows one or more sample images that matched in each comparison (S13). Note that the processing order of S12 and S13 is not limited to the example in Figure 5. S13 may be performed before S12, or S12 and S13 may be performed in parallel.

[0052] Next, using the flowchart in Figure 6, we will explain an example of the process flow for outputting aggregated information of selected sample images.

[0053] The processing unit 10 selects at least one sample image from among multiple sample images (S20). For example, the processing unit 10 may make this selection based on user input. Alternatively, the processing unit 10 may select at least one sample image from among multiple sample images according to any predetermined rule.

[0054] Next, the processing unit 10 outputs summary information of the selected sample images (S21).

[0055] <Effects and Effects> The processing unit 10 of this embodiment compares a search query with multiple sample images and searches for a sample image that matches the search query. The processing unit 10 then generates a matching history showing one or more sample images that matched in each matching operation. Based on this matching history, the processing unit 10 generates and outputs aggregated information regarding the matching results for the selected sample images.

[0056] Based on the aggregated information output, the user can understand an overview of the matching results for the selected sample image so far. Based on this understanding, the user can then decide whether to keep or delete the selected sample image. Furthermore, based on this understanding, the user can decide whether to keep, delete, or add variations of the selected sample image.

[0057] Variations of the selected sample image are images of the same type as the selected sample image. For example, if the selected sample image is an image containing a person in a seated position, then the variations of the selected sample image will also be images containing a person in a seated position. The selected sample image and its variations are not exactly identical to each other, and at least one characteristic differs between them. This characteristic may be, but is not limited to, the person's gender, age group, time of day the photo was taken, location of the photo, sitting position, or orientation.

[0058] Furthermore, the processing device 10 of this embodiment can generate and output characteristic aggregated information.

[0059] The aggregated information for selected sample images indicates, for example, the number of matches. If a selected sample image has a high number of matches, it can be considered to be of high importance. Conversely, if a selected sample image has a low number of matches, it can be considered to be of low importance. Furthermore, if a selected sample image has a low number of matches, it may be an outlier within the sample image group and could cause false positives. Therefore, if the number of matches is below a threshold, the selected sample image may be removed from the sample images.

[0060] Furthermore, aggregated information for selected sample images indicates, for example, the matching rate. A high matching rate suggests that the selected sample image is of high importance. Conversely, a low matching rate suggests that the selected sample image is of low importance. Additionally, a low matching rate indicates that the selected sample image is an outlier within the sample image group and may cause false detections. Therefore, if the matching rate is below a threshold, the selected sample image may be removed from the sample images.

[0061] Furthermore, the aggregated information for the selected sample image shows, for example, the statistical value of the number of other sample images that matched simultaneously. If there are many other sample images that matched simultaneously, it can be assumed that there are enough variations of that selected sample image registered. On the other hand, if there are too many other sample images that matched simultaneously, it can be assumed that there are an unnecessarily large number of variations of that selected sample image registered. In this case, some of the selected sample image and some of its variations may be deleted from the sample images.

[0062] Furthermore, if the number of other sample images that match simultaneously is small, it can be assumed that there are insufficient variations of the selected sample image. In this case, additional variations of the selected sample image may be registered. Additionally, if the number of other sample images that match simultaneously is small, the selected sample image may be an outlier in the sample image group, potentially causing false detections. Therefore, if the number of other sample images that match simultaneously is small, the selected sample image may be deleted from the sample images.

[0063] Furthermore, the aggregated information for the selected sample image includes, for example, other sample images that have been matched simultaneously. By viewing other sample images that have been matched simultaneously, it is possible to understand the variations of the selected sample image. If there are too many variations of the selected sample image, some of the selected sample image and some of the other sample images that have been matched simultaneously with it may be deleted from the sample images.

[0064] Furthermore, the aggregated information for the selected sample image indicates, for example, the number of times it has matched simultaneously with each other sample image that has matched at the same time. If this number is extremely high, it can be considered that the similarity between the other sample images and the selected sample image is too high, and having both registered in the sample image is redundant. For this reason, for example, either the other sample images or the selected sample image whose number of matches exceeds a threshold may be deleted from the sample image.

[0065] Furthermore, the aggregated information for selected sample images can indicate sample images that have been matched simultaneously with other sample images and that meet predetermined conditions based on the number of simultaneous matches. As mentioned above, these predetermined conditions include matching at or above a predetermined number of times, being within the top predetermined percentage, being within the top predetermined number, being within the standard rank, etc. If the number of simultaneous matches is extremely high, it can be considered that the similarity between the other sample images and the selected sample image is too high, and having both registered in the sample image is redundant. For this reason, for example, either the other sample images that meet the predetermined conditions or the selected sample image may be deleted from the sample images.

[0066] Furthermore, the aggregated information for the selected sample image indicates its similarity to other sample images that have been matched simultaneously. For example, either other sample images or the selected sample image whose similarity is above a certain threshold may be removed from the sample images.

[0067] <Third Embodiment> The processing unit 10 of this embodiment has a function to generate aggregated information that is different from the aggregated information described in the second embodiment. This will be described in detail below.

[0068] The search unit 12 searches for sample images whose similarity to the search query is equal to or greater than a certain threshold, and identifies them as sample images that match the search query. The search unit 12 then ranks the sample images that match the search query in descending order of similarity to the search query.

[0069] The registration unit 13 further registers the above ranking in the matching history, which shows one or more sample images that were matched in each matching round.

[0070] Figure 7 schematically shows an example of a matching history generated by the registration unit 13 of this embodiment. The matching history shown is similar to the matching history in Figure 3 described in the second embodiment, linking the matching number with the matched sample image.

[0071] The "Matching Number" field will contain the same information as the matching history shown in Figure 3, as described in the second embodiment.

[0072] The "Matched Sample Images" column stores the identification information of the sample images that matched the search query in each matching process. The numbers in parentheses shown indicate the ranking of the matched sample images, ordered from most similar to the search query.

[0073] The aggregation output unit 14 generates and outputs aggregation information regarding the selected sample images.

[0074] The aggregated information regarding the selected sample images includes, in addition to or instead of, the information described in the second embodiment, at least one of the following pieces of information. • Statistical values ​​of rankings when matching • Statistical values ​​of the ranking of each other sample image that have been matched simultaneously. • Statistical value of the rank difference between the rank at the time of matching and the ranks of other sample images that were matched at the same time. • History of the ranking pairs of other sample images that have been matched simultaneously with the selected sample image.

[0075] The "ranking statistics at the time of matching" are the ranking statistics of the selected sample images when they are matched. The ranking is assigned in order of similarity to the search query mentioned above. The statistics include the mean, maximum, minimum, median, mode, etc.

[0076] The "Statistical Rank of Each Other Sample Image That Has Matched Simultaneously" is a value calculated for each other sample image that has matched simultaneously with the selected sample image. This value is a statistical ranking of the other sample images that have matched simultaneously with the selected sample image. The ranking is assigned in order of similarity to the search query mentioned above. The statistical values ​​include the mean, maximum, minimum, median, mode, etc.

[0077] The "statistical value of the rank difference between the rank at the time of matching and the ranks of other sample images that have been matched simultaneously" is a value calculated for each other sample image that has been matched simultaneously with the selected sample image. The rank difference is the difference in rank between the two images when they were matched simultaneously.

[0078] In other words, the rank difference is the value obtained by subtracting the rank of the other sample images from the rank of the selected sample image when they were matched simultaneously. For example, if the rank of the selected sample image when they were matched simultaneously is 2nd and the rank of the other sample images is 4th, the rank difference is "-2".

[0079] Note that the rank difference may also be calculated by subtracting the rank of the selected sample image from the rank of the other sample images that were matched simultaneously. For example, if the selected sample image is ranked 2nd and the other sample images are ranked 4th when they were matched simultaneously, the rank difference is "2".

[0080] Alternatively, the rank difference may be the absolute value of the difference between the rank of the selected sample image and the rank of the other sample images when they are matched simultaneously. For example, if the selected sample image is ranked 2nd and the other sample images are ranked 4th when they are matched simultaneously, the rank difference is "2".

[0081] The ranking is based on the degree of similarity to the search query mentioned above, ordered from highest to lowest. Statistical values ​​include the mean, maximum, minimum, median, and mode.

[0082] The "History of Rank Pairs of Other Sample Images and Selected Sample Images that Have Matched Simultaneously" shows the history of pairs of rankings of other sample images and the selected sample image when they matched simultaneously. The rankings are assigned in order of similarity to the search query mentioned above.

[0083] The other configurations of the processing apparatus 10 in this embodiment are the same as those of the processing apparatus 10 in the first and second embodiments.

[0084] The processing apparatus 10 of this embodiment achieves the same effects and advantages as the processing apparatus 10 of the first and second embodiments.

[0085] Furthermore, the processing device 10 of this embodiment can generate and output characteristic aggregated information.

[0086] The aggregated information for the selected sample image shows, for example, the statistical value of the rank at the time of matching. It also shows, for example, the statistical value of the rank of each other sample image that has been matched simultaneously. Furthermore, it shows the statistical value of the rank difference between the selected sample image's rank at the time of matching and the rank of each other sample image that has been matched simultaneously. Finally, it shows the history of pairs of ranks between the selected sample image and other sample images that have been matched simultaneously.

[0087] A higher ranking in a match indicates a higher importance within the set of image variations. Conversely, a lower ranking indicates a lower importance within the set of image variations. Therefore, sample images with relatively low rankings in the set of image variations may be removed from the sample images.

[0088] <Fourth Embodiment> The processing unit 10 of this embodiment embodies a method for outputting aggregated information of selected sample images. This will be described in detail below.

[0089] The aggregation output unit 14 acquires the feature quantities of each of the multiple sample images. The feature quantities of each sample image are represented by multi-dimensional vectors. For example, the feature quantities of each sample image are represented by vectors of three or more dimensions.

[0090] The aggregation output unit 14 then reduces the dimensionality of the feature quantities of each sample image, which are represented by multi-dimensional vectors. For example, the aggregation output unit 14 reduces the dimensionality of the feature quantities of each sample image, which are represented by vectors of three or more dimensions, to two-dimensional vectors. The means of dimensionality reduction are not particularly limited, and methods such as t-SNE (t-Distributed Stochastic Neighbor Embedding) can be employed.

[0091] The aggregation output unit 14 then generates and outputs distribution information by plotting multiple sample images on a graph of the reduced dimension. An example of the distribution information is shown in Figure 8. In the example shown, multiple sample images are plotted on a two-dimensional graph. Note that in the figure, only the content of some of the sample images is shown, and only the outline of the other sample images is schematically shown. In reality, the content of all sample images can be shown. As another example, instead of sample images, marks such as circles or stars may be plotted on the graph.

[0092] The aggregation output unit 14 can accept input for selecting sample images on such a graph. Then, in response to this input, the aggregation output unit 14 can output aggregated information for the selected sample images, as shown in Figure 9. In Figure 9, the selected sample images are indicated using callouts. The aggregated information for the selected sample images is also indicated using these callouts.

[0093] The aggregated information of the selected sample images shown indicates the matching rate. The aggregated information of the selected sample images also indicates the average number of other sample images that matched simultaneously (the "average number of simultaneous matches" shown). Furthermore, the aggregated information of the selected sample images also indicates sample images that satisfy a predetermined condition (top four or fewer) in terms of the number of times they have matched simultaneously (the four sample images in the callouts in the figure). Additionally, the aggregated information of the selected sample images also indicates the number of times each of the other sample images that have matched simultaneously has matched simultaneously (the numbers next to the four sample images in the callouts in the figure). Note that the aggregated information of the selected sample images shown is an example. The aggregated output unit 14 may also display other aggregated information described in the second and third embodiments.

[0094] By the way, selecting each sample image from multiple sample images as shown in Figure 9 to check the aggregated information requires considerable effort. Therefore, the aggregation output unit 14 may highlight sample images that meet predetermined highlighting conditions from among the sample images plotted on the graph as shown in Figure 9. The user can then preferentially select the highlighted sample images and check the aggregated information.

[0095] There are no particular restrictions on the highlighting method. For example, you may surround only the sample image to be highlighted with a frame. Alternatively, you may make only the sample image to be highlighted blink. Alternatively, you may display the highlighted sample image larger than the other sample images.

[0096] The emphasis conditions are defined using the values ​​of predetermined items shown in the aggregated information. Examples of emphasis conditions include "number of matches is M or less" and "matching rate is N or less." Note that emphasis conditions may also be defined using the values ​​of multiple items shown in the aggregated information. Such emphasis conditions may include "number of matches is M or less, and the average number of other sample images that matched simultaneously is P or less."

[0097] The emphasis conditions may be pre-set. Alternatively, the user may set the emphasis conditions. That is, the aggregation output unit 14 may set the emphasis conditions based on user input.

[0098] The other configurations of the processing apparatus 10 in this embodiment are the same as those of the processing apparatus 10 in the first to third embodiments.

[0099] The processing apparatus 10 of this embodiment achieves the same effects and advantages as the processing apparatus 10 of the first to third embodiments.

[0100] Furthermore, the processing unit 10 of this embodiment can output aggregated information of selected sample images using a characteristic method as shown in Figures 8 and 9. Specifically, the processing unit 10 can reduce the dimensionality of each sample image's feature quantities, which are represented by multi-dimensional vectors, and then generate and output distribution information by plotting the multiple sample images on a graph of the reduced dimensions.

[0101] When distribution information is plotted on a multidimensional graph with multiple sample images, it becomes difficult for users to grasp the distribution of the multiple sample images. However, when the dimensionality is reduced and the distribution information is plotted on a graph of the reduced dimensions, users can easily grasp the distribution of the multiple sample images.

[0102] However, when distribution information is obtained by plotting multiple sample images on a graph of the reduced dimensions, it is possible to grasp the general distribution of the multiple sample images, but not the precise details. For example, even if multiple sample images appear close together on the graph, they may not actually be that close.

[0103] Therefore, as shown in Figure 9, the processing unit 10 of this embodiment displays aggregated information of selected sample images chosen from the sample images plotted on the graph. The user can make various decisions based on the distribution shown in the graph and the aggregated information of the selected sample images.

[0104] <Fifth Embodiment> The processing unit 10 of this embodiment outputs aggregated information of selected sample images using a method different from that of the fourth embodiment. This will be explained in detail below.

[0105] The aggregation output unit 14 displays multiple sample images with ranks based on the values ​​of predetermined items indicated in the aggregation information. The aggregation output unit 14 then accepts input to select a sample image from the multiple ranked sample images. The aggregation output unit 14 then outputs aggregation information related to the selected sample image.

[0106] Figure 10 shows an example. In Figure 10, multiple sample images are ranked in descending order of matching rate. Alternatively, the ranking may be based on values ​​from other items. Furthermore, users may be able to specify the items to be used for ranking.

[0107] By the way, in a rank display like the one shown in Figure 10, if the values ​​of all items shown in the aggregated information are displayed simultaneously, it becomes information overload and difficult to grasp the content. Therefore, in the example shown in Figure 10, the value of the first item among the multiple items shown in the aggregated information is displayed in the rank display. Then, depending on the selection of one sample image (selected sample image), the value of the second item among the multiple items shown in the aggregated information is displayed.

[0108] In other words, in the example shown in Figure 10, the aggregation output unit 14 displays the value of the first item shown in the aggregation information, linked to each of the multiple sample images in the rank display. In Figure 10, the matching rate and the average number of simultaneous matches are the first items. Note that the first item is not limited to this example.

[0109] In the example shown in Figure 10, the aggregation output unit 14 is configured to display the value of the second item shown in the aggregation information, according to the user's selection. In Figure 10, the second item is "a sample image that has been simultaneously matched with other sample images and satisfies a predetermined condition (within the top predetermined number)" (in the figure, "Top Simultaneous Detection"). Note that the second item is not limited to this example.

[0110] In Figure 10, one sample image is selected as the selected sample image by operating the "Top Simultaneous Detection" button. When the "Top Simultaneous Detection" button is operated, as shown in Figure 11, sample images that have been matched simultaneously with the selected sample image and satisfy a predetermined condition (within a predetermined number of top matching cases) are output.

[0111] Furthermore, in the example shown in Figure 10, by operating the "Distribution" button, the distribution information described in the fourth embodiment is output, which is obtained by reducing the dimensionality of each sample image's features, which are represented by multi-dimensional vectors, and then plotting the multiple sample images on a graph of the reduced dimensions. An example of the output distribution information is shown in Figure 12. In the example shown, the selected sample image, sample images that are easily matched simultaneously with the selected sample image, and other sample images are displayed separately.

[0112] Furthermore, the aggregation output unit 14 may also perform the highlighting described in the fourth embodiment, even in the rank display shown in Figure 10. The highlighting method is the same as that described in the fourth embodiment.

[0113] The other configurations of the processing apparatus 10 in this embodiment are the same as those of the processing apparatus 10 in the first to third embodiments.

[0114] The processing apparatus 10 of this embodiment achieves the same effects and advantages as the processing apparatus 10 of the first to third embodiments.

[0115] Furthermore, the processing unit 10 of this embodiment can output aggregated information of selected sample images using a characteristic method as shown in Figures 10 to 12. Specifically, the processing unit 10 can rank multiple sample images based on the values ​​of predetermined items indicated in the aggregated information. The processing unit 10 can then output aggregated information regarding the selected sample image chosen from among the ranked multiple sample images. Based on the outputted information, the user can grasp the overall picture of the multiple sample images and the details of each sample image.

[0116] <Variation> The processing unit 10 may also have a function to delete a sample image specified by the user from among the sample images stored in the storage device. For example, as shown in Figure 9, a button to delete the selected sample image may be displayed in the information (callout) associated with the selected sample image. Then, the processing unit 10 may delete the selected sample image from among the sample images stored in the storage device in response to the operation of this button.

[0117] Furthermore, in the rank display shown in Figure 10, the processing unit 10 may display a button to delete each sample image, linked to each sample image. The processing unit 10 may then delete the selected sample image from the sample images stored in the storage device in response to the operation of the button.

[0118] Furthermore, the processing unit 10 may also have a function to select and delete multiple sample images at once. For example, the processing unit 10 may accept user input to specify an area on a graph in the distribution information shown in Figures 8, 9, and 12. Then, in response to a subsequent deletion instruction from the user, the processing unit 10 may delete one or more sample images located within the specified area from the sample images stored in the storage device.

[0119] The embodiments of the present invention have been described above with reference to the drawings, but these are illustrative examples of the present invention, and various other configurations can be adopted. The configurations of the embodiments described above may be combined with each other, or some configurations may be replaced with other configurations. Furthermore, the configurations of the embodiments described above may be modified in various ways without departing from the spirit of the invention. In addition, the configurations and processes disclosed in each of the embodiments and modifications described above may be combined with each other.

[0120] Furthermore, the flowcharts used in the above description show multiple steps (processes) in sequence. However, the execution order of the steps performed in each embodiment is not limited to the order in which they are described. In each embodiment, the order of the illustrated steps can be changed to the extent that it does not impede the content. Also, the above embodiments can be combined to the extent that their content is not contradictory.

[0121] Some or all of the above embodiments may also be described as follows, but are not limited to the following. 1. Means of obtaining search queries, A search means that compares the search query with a plurality of sample images and searches for the sample image that matches the search query, A registration means for registering the results of the matching in a matching history that shows one or more of the sample images that were matched in each of the aforementioned matchings, Summary output means that generates and outputs summary information regarding the results of the comparison for a selected sample image from a plurality of sample images, based on the comparison history. A processing device. 2. The search means is, The sample images whose similarity to the search query is equal to or greater than a certain threshold are searched for as sample images that match the search query. The aggregated information relating to the selected sample images is as follows: The apparatus according to claim 1, which indicates at least one of the following: the number of matches, the matching rate, a statistical value of the number of other sample images that were matched simultaneously, other sample images that have been matched simultaneously, the number of times each of the other sample images that have been matched simultaneously was matched simultaneously, the sample image among the other sample images that have been matched simultaneously that satisfies a predetermined condition regarding the number of times it was matched simultaneously, and the similarity with other sample images that have been matched simultaneously. 3. The search means is, The sample images whose similarity to the search query is equal to or greater than a certain threshold are searched for as sample images that match the search query, and the sample images that match the search query are ranked in descending order of similarity. The aggregated information relating to the selected sample images is as follows: The processing apparatus according to 1 or 2, which shows at least one of the following: statistical values ​​of the rank at the time of matching; statistical values ​​of the rank at the time of matching for each of the other sample images that have been matched at the same time; statistical values ​​of the rank difference between the rank at the time of matching and the rank at the time of matching for each of the other sample images that have been matched at the same time; and a history of pairs of ranks of the other sample images and the selected sample image that have been matched at the same time. 4. The aggregate output means is: After reducing the dimensionality of the feature quantities of each sample image, which are represented by multi-dimensional vectors, distribution information is generated and output by plotting the multiple sample images on a graph of the reduced dimensions. On the aforementioned graph, input is received to select the sample image to be selected. A processing device according to any one of 1 to 3 that outputs the aggregated information relating to the selected sample image. 5. The aggregate output means is: The processing apparatus according to 4, which highlights the sample images plotted on the graph, the sample images whose values ​​in the aggregated information satisfy the highlighting conditions. 6. The aggregate output means is: The processing device according to 5, which sets the emphasis conditions based on user input. 7. The aggregate output means is: Multiple sample images are ranked and displayed based on the values ​​of predetermined items shown in the aggregated information. The system accepts input to select a sample image from among the multiple sample images that are ranked as described above. A processing device according to any one of 1 to 6 that outputs the aggregated information relating to the selected sample image. 8. The aggregate output means is: In the aforementioned rank display, the value of the first item shown in the aggregated information is displayed, linked to each of the multiple sample images. The processing device according to 7, which displays the value of the second item shown in the aggregate information as aggregate information relating to the selected sample image. 9. One or more computers, Get the search query, The search query is compared with multiple sample images, and the sample images that match the search query are searched for. The results of the matching are registered in the matching history, which shows one or more of the sample images that were matched in each of the aforementioned matching attempts. A processing method for generating and outputting aggregated information regarding the results of the comparison for a selected sample image from among multiple sample images, based on the comparison history. 10. Computers, A means of obtaining search queries. A search means that compares the search query with a plurality of sample images and searches for the sample image that matches the search query. A registration means for registering the results of the matching in a matching history that shows one or more of the sample images that were matched in each of the aforementioned matchings, Aggregation output means that generates and outputs aggregated information regarding the matching results for a selected sample image from a plurality of sample images based on the matching history, A program that makes it function as such.

[0122] This application claims priority based on Japanese Patent Application No. 2022-207926, filed on 26 December 2022, and incorporates all of its disclosures herein. [Explanation of symbols]

[0123] 10 Processing Unit 11 Acquisition Department 12 Search Section 13 Registration Department 14. Output section for aggregation 1A Processor 2A Memory 3A input / output I / F 4A Peripheral Circuits 5A bus

Claims

1. Methods for obtaining search queries, A search means that compares the search query with a plurality of sample images and searches for the sample image that matches the search query, A registration means for registering the results of the matching in a matching history that shows one or more of the sample images that were matched in each of the aforementioned matchings, Summary output means that generates and outputs summary information regarding the results of the comparison for a selected sample image from a plurality of sample images, based on the comparison history. A processing device.

2. The search means is, The sample images whose similarity to the search query is equal to or greater than a certain threshold are searched for as sample images that match the search query. The aggregated information relating to the selected sample images is as follows: The apparatus according to claim 1, which provides at least one of the following: the number of matches, the matching rate, a statistical value of the number of other sample images that were matched simultaneously, other sample images that have been matched simultaneously, the number of times each of the other sample images that have been matched simultaneously was matched simultaneously, the sample image among the other sample images that have been matched simultaneously that satisfies a predetermined condition regarding the number of times it was matched simultaneously, and the similarity with other sample images that have been matched simultaneously.

3. The search means is, The sample images whose similarity to the search query is equal to or greater than a certain threshold are searched for as sample images that match the search query, and the sample images that match the search query are ranked in descending order of similarity. The aggregated information relating to the selected sample images is as follows: The apparatus according to claim 1 or 2, which provides at least one of the following: statistical values ​​of the rank at the time of matching; statistical values ​​of the rank at the time of matching for each of the other sample images that have been matched at the same time; statistical values ​​of the rank difference between the rank at the time of matching and the rank at the time of matching for each of the other sample images that have been matched at the same time; and a history of pairs of ranks between other sample images and the selected sample image that have been matched at the same time.

4. The aforementioned aggregation output means is After reducing the dimensionality of the feature quantities of each sample image, which are represented by multi-dimensional vectors, distribution information is generated and output by plotting the multiple sample images on a graph of the reduced dimensions. On the aforementioned graph, input is received to select the sample image to be selected. The processing apparatus according to claim 1 or 2, which outputs the aggregated information relating to the selected sample image.

5. The aforementioned aggregation output means is The processing apparatus according to claim 4, which highlights the sample images plotted on the graph in which the value of a predetermined item indicated by the aggregated information satisfies the highlighting condition.

6. The aforementioned aggregation output means is The processing apparatus according to claim 1 or 2, which identifies a plurality of sample images that have been matched simultaneously based on the matching history.

7. The aforementioned aggregation output means is Multiple sample images are ranked and displayed based on the values ​​of predetermined items shown in the aggregated information. The system accepts input to select a sample image from among the multiple sample images that are ranked as described above. The processing apparatus according to claim 1 or 2, which outputs the aggregated information relating to the selected sample image.

8. The aforementioned aggregation output means is In the aforementioned rank display, the value of the first item shown in the aggregated information is displayed in association with each of the multiple sample images. The processing apparatus according to claim 7, which displays the value of the second item shown in the aggregated information as aggregated information relating to the selected sample image.

9. One or more computers, Get the search query, The search query is compared with multiple sample images, and the sample images that match the search query are searched for. The results of the matching are registered in the matching history, which shows one or more of the sample images that were matched in each of the aforementioned matching attempts. A processing method for generating and outputting aggregated information regarding the results of the comparison for a selected sample image from among multiple sample images, based on the comparison history.

10. Computers, A means of obtaining search queries. A search means that compares the search query with a plurality of sample images and searches for the sample image that matches the search query. A registration means for registering the results of the matching in a matching history that shows one or more of the sample images that were matched in each of the aforementioned matchings, Aggregation output means that generates and outputs aggregated information regarding the matching results for a selected sample image from a plurality of sample images based on the matching history, A program that makes it function as such.

Images