Processing device, processing method, and program

Abstract

The present invention provides a processing device comprising: an acquisition unit that acquires a search query; a search unit that matches the search query with a plurality of sample images and searches for a sample image that matches the search query; a registration unit that registers the matching results in a matching history indicating one or a plurality of sample images matched in each matching; and an aggregation output unit that generates and outputs aggregation information regarding the matching results for a selected sample image among the plurality of sample images on the basis of the matching history.

Description

Processing device, processing method, and recording medium

[0001] The present invention relates to a processing device, a processing method, and a program.

[0002] Techniques related to the present invention are disclosed in Patent Documents 1 and 2. The techniques disclosed in Patent Documents 1 and 2 create a database that stores images of people in various poses, and then search the database for images that match a search query specified by a user and output the images.

[0003] International Publication No. 2021 / 084677 Japanese Patent Application Laid-Open No. 2019-091138

[0004] In technology that searches a database for images that match a search query and outputs them, it is important to consider what kind of images to register in the database.

[0005] Registering a large number of diverse images in a database increases the probability of finding an image that matches a search query. For example, registering images of people in various postures (e.g., sitting, standing, etc.) in a database increases the probability of finding an image that matches a search query. Also, registering multiple variations of an image in a database, rather than just one image of a certain type (e.g., an image of a person in a sitting posture), increases the probability of finding an image that matches a search query.

[0006] On the other hand, the more images are registered in the database, the larger the storage capacity required to store the images, which results in increased costs and labor required for data management.

[0007] Taking these points into consideration, it is necessary to optimize the set of images to be registered in the database. Optimization involves, for example, deleting redundant images from the database and adding missing images to the database.

[0008] However, a certain number of images are registered in a database. For this reason, it is not easy to grasp the overall picture of the registered images and identify which images are redundant and which images are missing. Therefore, there is a need for technology to support the optimization of the images registered in the database.

[0009] The techniques disclosed in Patent Documents 1 and 2 are both techniques for searching a database for images that match a search query and outputting them, but neither Patent Document 1 nor 2 can support optimization of the group of images registered in the database.

[0010] In view of the above-mentioned problems, one example of the object of the present invention is to provide a processing device, a processing method, and a program that solve the problem of supporting the optimization of a group of images registered in a database in a technology for searching a database for images that match a search query and outputting the images.

[0011] According to one aspect of the present invention, there is provided a processing device having: an acquisition means for acquiring a search query; a search means for matching the search query with a plurality of sample images and searching for the sample images that match the search query; a registration means for registering the results of the matching in a matching history that indicates one or more of the sample images that match in each matching; and a summary output means for generating and outputting summary information regarding the results of the matching for a selected sample image from the plurality of sample images based on the matching history.

[0012] According to one aspect of the present invention, a processing method is provided in which one or more computers acquire a search query, match the search query with a plurality of sample images, search for sample images that match the search query, register the results of the matching in a matching history that indicates one or more sample images that match in each matching, and generate and output aggregated information regarding the results of the matching for selected sample images from the plurality of sample images based on the matching history.

[0013] According to one aspect of the present invention, there is provided a program that causes a computer to function as: an acquisition means that acquires a search query; a search means that compares the search query with a plurality of sample images and searches for the sample images that match the search query; a registration means that registers the results of the comparison in a comparison history that indicates one or more sample images that match in each comparison; and a summary output means that generates and outputs summary information regarding the results of the comparison for a selected sample image from the plurality of sample images based on the comparison history.

[0014] According to one aspect of the present invention, in a technology for searching a database for images that match a search query and outputting them, a processing device, a processing method, and a program are realized that solve the problem of supporting the optimization of a group of images registered in a database.

[0015] The above-mentioned objects and other objects, features and advantages will become more apparent from the following description of the preferred embodiments and the accompanying drawings.

[0016] FIG. 1 is a diagram showing an example of a functional block diagram of a processing device. FIG. 2 is a diagram showing an example of a hardware configuration of a processing device. FIG. 3 is a diagram showing an example of a verification history generated by a processing device. FIG. 4 is a diagram showing an example of aggregate information generated by a processing device. FIG. 5 is a flowchart showing an example of a processing flow of a processing device. FIG. 6 is a flowchart showing another example of a processing flow of a processing device. FIG. 7 is a diagram showing an example of a verification history generated by a processing device. FIG. 8 is a diagram showing an example of information output by a processing device. FIG. 9 is a diagram showing an example of information output by a processing device. FIG. 10 is a diagram showing an example of information output by a processing device.

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0018] 1 is a functional block diagram showing an overview of a processing device 10 according to a first embodiment. The processing device 10 includes an acquisition unit 11, a search unit 12, a registration unit 13, and a tally output unit 14.

[0019] The acquisition unit 11 acquires a search query. The search unit 12 compares the search query with a plurality of sample images and searches for sample images that match the search query. The registration unit 13 registers the results of the comparison in a comparison history that indicates one or more sample images that match in each comparison. The summary output unit 14 generates and outputs summary information regarding the results of the comparison for a selected sample image from the plurality of sample images based on the comparison history.

[0020] In this way, the processing device 10 performs a search process to search a database for sample images that match the search query. The processing device 10 then registers the results of matching the search query and sample images in each search process as a matching history. The processing device 10 uses this matching history to generate and output aggregated information regarding the matching results for each sample image. The aggregated information indicates, for example, the number of times matching occurred in past search processes. Based on this aggregated information, the user can identify which images are redundant and which images are missing.

[0021] In this way, the processing device 10 of this embodiment solves the problem of supporting the optimization of the group of images registered in a database in a technology that searches a database for and outputs images that match a search query.

[0022] Second Embodiment "Overview" The processing device 10 of this embodiment is a specific implementation of the processing device 10 of the first embodiment. In this embodiment, the configuration of the processing device 10 is specific. In addition, in this embodiment, the "aggregated information" described in the first embodiment is specific.

[0023] "Hardware Configuration" Next, an example of the hardware configuration of the processing device 10 will be described. Each functional unit of the processing device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. Software includes programs that are pre-loaded in the device before shipping, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0024] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a processing device 10. As shown in FIG. 2, the processing device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing device 10 does not necessarily have to have the peripheral circuit 4A. Note that the processing device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices may have the above hardware configuration.

[0025] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a processing unit such as a CPU or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, touch panel, etc. Examples of output devices include a display, speaker, printer, mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0026] "Functional Configuration" Next, the functional configuration of the processing device 10 of this embodiment will be described in detail. Fig. 1 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 has an acquisition unit 11, a search unit 12, a registration unit 13, and a tally output unit 14.

[0027] The acquisition unit 11 acquires a search query. The search query is, for example, a query image. The query image may be a still image or a moving image. Alternatively, the search query may be data indicating the features of the query image in text or the like. The features of the query image are features of an object appearing in the query image. The objects appearing in the query image include, but are not limited to, people, other animals, buildings, natural objects, man-made objects, etc. The features of the objects appearing in the query image include, but are not limited to, facial features, body features, clothing features, belongings features, posture features, gait features, movement features, movement trajectories, size, position, color, shape, feature point information, etc.

[0028] Here, an example of a means by which the acquisition unit 11 acquires a search query will be described. For example, a user may input a search query via an input device of the processing device 10. Then, the acquisition unit 11 may acquire the input search query. Alternatively, the processing device 10 may be a server. Then, the acquisition unit 11 may acquire a search query transmitted from a client terminal. Alternatively, an image generated by a surveillance camera may be input to the processing device 10. Then, the acquisition unit 11 may acquire the image generated by the surveillance camera as a search query. Note that the means by which the acquisition unit 11 acquires a search query is not limited to the example shown here.

[0029] The search unit 12 searches for sample images that match the search query acquired by the acquisition unit 11.

[0030] A plurality of sample images according to the purpose of the search are stored in advance in the storage device. For example, if the purpose of the search is to "search for images showing a person in a pose indicated by a search query," sample images of people in various poses are stored in the storage device. Also, if the purpose of the search is to "identify the shooting position of an image," sample images taken in various positions are stored in the storage device. Note that the processing device 10 may include the above storage device. Alternatively, an external device configured to be able to communicate with the processing device 10 may include the above storage device.

[0031] The search unit 12 compares the search query acquired by the acquisition unit 11 with each of the multiple sample images and calculates the similarity. Then, the search unit 12 identifies a sample image that matches the search query based on the calculated similarity. The condition for matching with the search query is that the similarity is equal to or greater than a reference value. Note that the comparison and calculation of the similarity can be achieved using any conventional technology. For example, the technology described in Patent Document 1 can be used.

[0032] The registration unit 13 registers the results of the matching performed by the search unit 12 in a matching history indicating one or more sample images matched in each matching.

[0033] An example of the matching history is shown in Fig. 3. The matching history shown in the figure associates the matching number with the matched sample image.

[0034] The "verification number" is information that distinguishes between multiple verifications that have been performed up to now.

[0035] The "Matched Sample Image" column registers the identification information of the sample image that matched the search query in each comparison.

[0036] As mentioned above, the condition for matching with a search query is that the degree of similarity must be equal to or greater than a certain threshold. Therefore, as shown in Figure 3, multiple sample images may be matched simultaneously in each comparison.

[0037] Returning to FIG. 1, the tally output unit 14 generates and outputs tally information on the results of matching for a selected sample image from among a plurality of sample images based on the matching history.

[0038] The selected sample image is at least one sample image selected from a plurality of sample images. The user may make the selection. Alternatively, the tally output unit 14 may make the selection according to any predetermined rule. In the following embodiment, an example in which the user makes the selection will be described in detail.

[0039] The aggregate information about the selected sample image indicates at least one of the following information: Number of matches; Matching rate; Statistics on the number of other sample images that have been matched at the same time; Other sample images that have been matched at the same time; Number of times that each of the other sample images that have been matched at the same time has been matched at the same time; Sample images among the other sample images that have been matched at the same time that have been matched at the same time satisfy a predetermined condition; Similarity with other sample images that have been matched at the same time

[0040] The "number of matches" is the number of matches with the search query in the multiple matches shown in the matching history shown in Fig. 3. The number of matches is calculated, for example, by counting the number of times the identification information of the selected sample image is registered in the "matched sample image" column of the matching history shown in Fig. 3.

[0041] The "matching rate" is calculated by dividing the number of times the above-mentioned matching has occurred by the total number of times the matching has occurred indicated in the matching history shown in FIG.

[0042] The "statistical value of the number of other sample images that were matched at the same time" is a statistical value of the number of other sample images that were matched at the same time when the selected sample image was matched. The statistical value may be the average, maximum, minimum, median, mode, etc. As described above, the condition for matching with a search query is "similarity equal to or greater than a reference value." Therefore, as shown in Figure 3, multiple sample images may be matched at the same time in each matching.

[0043] "Other sample images that have been matched at the same time" are sample images that have been matched at the same time as the selected sample image.

[0044] "Number of times simultaneously matched with each other sample image that has been simultaneously matched with the selected sample image" indicates the number of times simultaneously matched with the selected sample image for each sample image that has been simultaneously matched with the selected sample image.

[0045] "Sample images among other sample images that have been simultaneously matched, whose number of simultaneous matches satisfies a predetermined condition" are sample images whose number of simultaneous matches with the selected sample image satisfies a predetermined condition. The predetermined condition is, for example, a predetermined number of times or more. Alternatively, the other sample images may be ranked in descending order of the number of simultaneous matches with the selected sample image. The predetermined condition may be within a predetermined percentage of the rank, within a predetermined number of the top images, or within a reference ranking.

[0046] The "similarity to other sample images that have been matched at the same time" is the similarity between the selected sample image and each of the other sample images that have been matched at the same time with the selected sample image. The calculation of the similarity between two images can be realized using any conventional technology.

[0047] The tally output unit 14 generates tally information regarding the selected sample images as described above based on the matching history as shown in FIG. 3. The tally output unit 14 then outputs the generated tally information regarding the selected sample images. For example, the tally output unit 14 may output the tally information via an output device such as a display, a projection device, or a printer. Alternatively, the processing device 10 may be a server. The tally output unit 14 may then transmit the tally information to a client terminal. In this case, the output device of the client terminal outputs the tally information.

[0048] The tally output unit 14 may generate and output tally information about a selected sample image each time the selected sample image is selected from the plurality of sample images. Alternatively, the tally output unit 14 may generate tally information about each of the plurality of sample images in advance based on the latest matching history, as shown in Fig. 4. Then, when the selected sample image is selected from the plurality of sample images, the tally output unit 14 may extract and output the tally information about the selected sample image from the tally information about each of the plurality of sample images.

[0049] Next, an example of the processing flow of the processing device 10 will be described with reference to the flowcharts of FIGS.

[0050] First, an example of the flow of processing for generating a collation history as shown in FIG. 3 will be described with reference to the flowchart of FIG.

[0051] When the processing device 10 acquires a search query (S10), it compares the search query with a plurality of sample images and searches for sample images that match the search query (S11). Next, the processing device 10 outputs a search result indicating the matched sample images (S12). The processing device 10 also registers the result of the comparison in S11 in a comparison history indicating one or more sample images that matched in each comparison (S13). The order of processing S12 and S13 is not limited to the example in FIG. 5 . S13 may be performed before S12, or S12 and S13 may be performed in parallel.

[0052] Next, an example of the flow of processing for outputting the summary information of selected sample images will be described with reference to the flowchart of FIG.

[0053] The processing device 10 selects at least one sample image from the plurality of sample images (S20). For example, the processing device 10 may perform the selection based on a user input. Alternatively, the processing device 10 may select at least one sample image from the plurality of sample images according to any predetermined rule.

[0054] Next, the processing device 10 outputs the summary information of the selected sample images (S21).

[0055] <Effects> The processing device 10 of this embodiment compares a search query with multiple sample images and searches for sample images that match the search query. The processing device 10 then generates a comparison history that indicates one or more sample images that matched in each comparison. Based on this comparison history, the processing device 10 generates and outputs summary information regarding the results of comparison for selected sample images.

[0056] Based on the output summary information, the user can grasp an overview of the results of matching the selected sample image up to now. Based on the grasped content, the user can then decide whether to keep the selected sample image as a sample image or delete it. Based on the grasped content, the user can also decide whether to keep, delete, or add a variation image of the selected sample image.

[0057] The image of a variation of the selected sample image is an image of the same type as the selected sample image. For example, if the selected sample image is an image containing a person in a seated position, the image of the variation of the selected sample image is also an image containing a person in a seated position. The selected sample image and the image of the variation of the selected sample image do not completely match each other, but differ from each other in at least one characteristic. Examples of the characteristic include, but are not limited to, the person's gender, age group, time of day, location, sitting position, and orientation.

[0058] Furthermore, the processing device 10 of this embodiment can generate and output characteristic aggregate information.

[0059] The aggregate information of the selected sample image indicates, for example, the number of matches. If the number of matches is high, the selected sample image can be considered to have high importance. If the number of matches is low, the selected sample image can be considered to have low importance. If the number of matches is low, the selected sample image is an outlier among the sample images and may cause false detection. Therefore, if the number of matches is below a threshold, the selected sample image may be deleted from the sample images.

[0060] Furthermore, the aggregate information of the selected sample images indicates, for example, the matching rate. If the matching rate is high, the selected sample image can be considered to have high importance. If the matching rate is low, the selected sample image can be considered to have low importance. If the matching rate is low, the selected sample image is an outlier among the group of sample images, and may cause erroneous detection. Therefore, if the matching rate is below a threshold, the selected sample image may be deleted from the sample images.

[0061] Furthermore, the aggregate information for the selected sample image indicates, for example, a statistical value of the number of other sample images that were matched at the same time. If a large number of other sample images were matched at the same time, it can be assumed that a sufficient number of variation images of the selected sample image have been registered. On the other hand, if too many other sample images were matched at the same time, it can be assumed that an unnecessarily large number of variation images of the selected sample image have been registered. In this case, it is possible to delete the selected sample image and some of the variation images of the selected sample image from the sample images.

[0062] Furthermore, if the number of other sample images that are simultaneously matched is small, it can be assumed that there are insufficient variation images of the selected sample image. Therefore, variation images of the selected sample image may be additionally registered. Furthermore, if the number of other sample images that are simultaneously matched is small, the selected sample image may be an outlier among the sample images, which may cause false detection. Therefore, if the number of other sample images that are simultaneously matched is small, the selected sample image may be deleted from the sample images.

[0063] Furthermore, the aggregate information of the selected sample image indicates, for example, other sample images that have been matched at the same time. By viewing the other sample images that have been matched at the same time, it is possible to understand the variations of the selected sample image. If there are too many variations of the selected sample image, the selected sample image and some of the other sample images that have been matched at the same time with the selected sample image may be deleted from the sample images.

[0064] Furthermore, the aggregate information of the selected sample image indicates, for example, the number of times it has been simultaneously matched with each of the other sample images that it has been simultaneously matched with. If the number of times is extremely high, it can be considered that the similarity between the other sample images and the selected sample image is too high, and that having both registered as sample images is redundant. Therefore, for example, either the other sample images or the selected sample image whose number of times is equal to or exceeds a threshold may be deleted from the sample images.

[0065] Furthermore, the aggregate information for the selected sample image can indicate sample images that have been simultaneously matched with other sample images and whose simultaneous matching frequency meets a predetermined condition. As described above, the predetermined condition can be a predetermined number of times or more, within a predetermined top percentage, within a predetermined top number, within a reference ranking, etc. If the simultaneous matching frequency is extremely high, the similarity between the other sample image and the selected sample image is too high, and it can be considered that both are redundantly registered as sample images. Therefore, for example, either the selected sample image or the other sample image that meets the predetermined condition can be deleted from the sample images.

[0066] The aggregate information of the selected sample image also indicates the degree of similarity with other sample images that have been matched at the same time. For example, either the selected sample image or other sample images whose degree of similarity is equal to or exceeds a threshold may be deleted from the sample images.

[0067] Third Embodiment The processing device 10 of this embodiment has a function of generating tally information different from the tally information described in the second embodiment, which will be described in detail below.

[0068] The search unit 12 searches for sample images whose similarity to the search query is equal to or greater than a reference value as sample images that match the search query, and then ranks the sample images that match the search query in descending order of similarity to the search query.

[0069] The registration unit 13 further registers the ranking in a matching history indicating one or more sample images matched in each matching.

[0070] 7 is a diagram illustrating an example of a matching history generated by the registration unit 13 of this embodiment. The illustrated matching history associates a matching number with a matching sample image, similar to the matching history of FIG. 3 described in the second embodiment.

[0071] In the "verification number" field, the same content as the verification history in FIG. 3 described in the second embodiment is registered.

[0072] The "Matched Sample Image" column registers the identification information of the sample image that matched the search query in each comparison. The parenthesized numbers shown in the figure indicate the ranking of the sample images that matched the search query in order of their similarity to the search query.

[0073] The tally output unit 14 generates and outputs tally information regarding the selected sample images.

[0074] The aggregate information regarding the selected sample image indicates at least one of the following information in addition to or instead of the information described in the second embodiment: Statistics of the ranking when matched Statistics of the ranking when each of other sample images that have been matched at the same time Statistics of the difference between the ranking when matched and the ranking when each of other sample images that have been matched at the same time History of pairs of rankings of other sample images that have been matched at the same time and the selected sample image

[0075] "Statistical values ​​of ranking when matching occurs" are statistical values ​​of the ranking of the selected sample image when the selected sample image matches. The ranking is determined in descending order of similarity to the search query described above. The statistical values ​​include the average, maximum, minimum, median, and mode.

[0076] The "statistical value of the ranking of each other sample image that has been matched at the same time" is a value calculated for each other sample image that has been matched at the same time with the selected sample image. This value is a statistical value of the ranking when matched at the same time with the selected sample image. The ranking is determined in descending order of similarity to the search query described above. The statistical value may be the average, maximum, minimum, median, mode, etc.

[0077] The "statistical value of the ranking difference between the ranking at the time of matching and the ranking of each other sample image that has been matched at the same time" is a value calculated for each other sample image that has been matched at the same time as the selected sample image. The ranking difference is the difference between the rankings of each other when they were matched at the same time.

[0078] That is, the rank difference is the value obtained by subtracting the rank of the other sample image from the rank of the selected sample image when they were simultaneously matched. For example, if the rank of the selected sample image when they were simultaneously matched was second and the rank of the other sample image was fourth, the rank difference is "-2".

[0079] The ranking difference may be a value obtained by subtracting the ranking of the selected sample image from the ranking of the other sample images when they were matched simultaneously. For example, if the ranking of the selected sample image when they were matched simultaneously was second and the ranking of the other sample image was fourth, the ranking difference is "2".

[0080] Alternatively, the rank difference may be the absolute value of the difference between the rank of the selected sample image and the rank of another sample image when they are simultaneously matched. For example, if the rank of the selected sample image when they are simultaneously matched is second and the rank of another sample image is fourth, the rank difference is "2".

[0081] The ranking is determined in descending order of similarity to the search query. The statistical values ​​include the average, maximum, minimum, median, and mode.

[0082] "History of pairs of rankings of other sample images and the selected sample image that have been matched at the same time" shows the history of pairs of rankings of other sample images and the selected sample image when they were matched at the same time. The rankings are ranked in descending order of similarity to the search query described above.

[0083] Other configurations of the processing apparatus 10 of this embodiment are similar to those of the processing apparatus 10 of the first and second embodiments.

[0084] According to the processing apparatus 10 of this embodiment, the same effects as those of the processing apparatus 10 of the first and second embodiments are achieved.

[0085] Furthermore, the processing device 10 of this embodiment can generate and output characteristic aggregate information.

[0086] The aggregated information of the selected sample image indicates, for example, statistical values ​​of the ranking when matched. The aggregated information of the selected sample image also indicates, for example, statistical values ​​of the ranking when each of other sample images that have been matched at the same time were matched. The aggregated information of the selected sample image also indicates statistical values ​​of the ranking difference between the ranking when matched and the ranking when each of other sample images that have been matched at the same time were matched. The aggregated information of the selected sample image also indicates the history of pairs of rankings of the selected sample image and other sample images that have been matched at the same time.

[0087] The higher the ranking when matching, the more important the image in the group of variation images can be considered to be. The lower the ranking when matching, the less important the image in the group of variation images can be considered to be. Therefore, a sample image that has a relatively low ranking when matching in the group of variation images can be deleted from the sample images.

[0088] Fourth Embodiment The processing device 10 of this embodiment embodies a method for outputting aggregate information of selected sample images, which will be described in detail below.

[0089] The tallying and outputting unit 14 acquires the feature quantities of each of the plurality of sample images. The feature quantities of each sample image are represented by multi-dimensional vectors. For example, the feature quantities of each sample image are represented by three or more dimensional vectors.

[0090] The tallying and outputting unit 14 then reduces the dimension of the feature quantities of each sample image represented by a multi-dimensional vector. For example, the tallying and outputting unit 14 reduces the dimension of the feature quantities of each sample image represented by a three- or more-dimensional vector to a two-dimensional vector. There are no particular limitations on the means of dimension reduction, and a technique such as t-SNE (t-Distributed Stochastic Neighbor Embedding) can be used.

[0091] The tallying and outputting unit 14 then generates and outputs distribution information in which multiple sample images are plotted on a graph of the reduced dimensions. FIG. 8 shows an example of the distribution information. In the illustrated example, multiple sample images are plotted on a two-dimensional graph. Note that in the figure, the image contents of only some of the sample images are shown, and for the other sample images, only the outlines of the images are shown schematically. In reality, the image contents can be shown for all of the sample images. As another example, instead of sample images, marks such as circles or stars may be plotted on the graph.

[0092] The tally output unit 14 can accept an input for selecting a selected sample image on such a graph. Then, in response to the input, the tally output unit 14 can output tally information for the selected selected sample image, as shown in Fig. 9. In Fig. 9, the selected selected sample image is shown using a speech bubble. The tally information for the selected sample image is also shown using the speech bubble.

[0093] The illustrated aggregate information for the selected sample images indicates the matching rate. The illustrated aggregate information for the selected sample images also indicates the average number of other sample images that were matched simultaneously (the "average number of simultaneous matches" shown in the figure). The illustrated aggregate information for the selected sample images also indicates sample images whose number of simultaneous matches meets a predetermined condition (within the top four) (the four sample images in the speech bubbles in the figure). The illustrated aggregate information for the selected sample images also indicates the number of simultaneous matches with each of the other sample images that have been matched simultaneously (the numbers next to the four sample images in the speech bubbles in the figure). Note that the illustrated aggregate information for the selected sample images is just an example. The aggregate output unit 14 may also display other aggregate information described in the second and third embodiments.

[0094] However, it requires a great deal of effort to select each sample image one by one from among the multiple sample images shown in Fig. 9 and check the tabulated information. Therefore, the tabulated output unit 14 may highlight sample images that satisfy predetermined highlighting conditions among the sample images plotted on the graph as shown in Fig. 9. The user can preferentially select the highlighted sample images and check the tabulated information.

[0095] The highlighting method is not particularly limited. For example, only the sample image to be highlighted may be surrounded by a frame. Alternatively, only the sample image to be highlighted may be made to blink. Alternatively, the sample image to be highlighted may be displayed larger than the other sample images.

[0096] The emphasis condition is defined using the values ​​of predetermined items indicated in the aggregate information. Examples of the emphasis condition include "the number of matches is M or less" and "the matching rate is N or less." The emphasis condition may also be defined using the values ​​of multiple items indicated in the aggregate information. An example of such an emphasis condition is "the number of matches is M or less, and the average number of other sample images matched simultaneously is P or less."

[0097] The highlighting conditions may be set in advance, or the user may set the highlighting conditions. That is, the tally output unit 14 may set the highlighting conditions based on user input.

[0098] Other configurations of the processing apparatus 10 of this embodiment are similar to those of the processing apparatus 10 of the first to third embodiments.

[0099] According to the processing apparatus 10 of this embodiment, the same effects as those of the processing apparatus 10 of the first to third embodiments are achieved.

[0100] Furthermore, the processing device 10 of this embodiment can output aggregate information of selected sample images using a characteristic method as shown in Figures 8 and 9. Specifically, the processing device 10 can reduce the dimensions of the feature amounts of each sample image represented by a multi-dimensional vector, and then generate and output distribution information in which the multiple sample images are plotted on a graph of the reduced dimensions.

[0101] In the case of distribution information in which multiple sample images are plotted on a multidimensional graph, it is difficult for users to grasp the distribution status of the multiple sample images. In the case of distribution information in which multiple sample images are plotted on a graph of reduced dimensions after dimensional reduction, users can easily grasp the distribution status of the multiple sample images.

[0102] However, in the case of distribution information in which multiple sample images are plotted on a graph of the reduced dimension, although it is possible to grasp the general distribution of the multiple sample images, it is not possible to grasp the details accurately. For example, even if the distance between multiple sample images appears close to each other on the graph, it may occur that they are not actually that close.

[0103] Therefore, the processing device 10 of this embodiment displays the aggregate information of selected sample images selected from the sample images plotted on a graph, as shown in Fig. 9. The user can make various decisions based on the distribution status shown in the graph and the aggregate information of the selected sample images.

[0104] Fifth Embodiment The processing device 10 of this embodiment outputs the summary information of the selected sample images using a method different from that of the fourth embodiment, which will be described in detail below.

[0105] The tally output unit 14 ranks the sample images based on the values ​​of the predetermined items indicated by the tally information. The tally output unit 14 then accepts an input to select a selected sample image from the ranked sample images. The tally output unit 14 then outputs tally information related to the selected sample image.

[0106] An example is shown in Figure 10. In Figure 10, multiple sample images are ranked in descending order of matching rate. The ranking may also be based on values ​​of other items. The user may also be able to specify the items to be used for ranking.

[0107] However, if the values ​​of all items shown in the aggregate information were displayed simultaneously in a rank display such as that shown in Figure 10, the amount of information would be excessive, making it difficult to understand the content. Therefore, in the example shown in Figure 10, the value of the first item among the multiple items shown in the aggregate information is displayed in the rank display. Then, in response to the selection of one sample image (selected sample image), the value of the second item among the multiple items shown in the aggregate information is displayed.

[0108] 10, the tally output unit 14 displays the value of the first item indicated in the tally information in association with each of the plurality of sample images in the rank display. In FIG. 10, the matching rate and the average number of simultaneous matches are the first items. Note that the first item is not limited to this example.

[0109] 10, the tally output unit 14 is configured to display the value of the second item indicated in the tally information in response to a user selection. In FIG. 10, the second item is "a sample image whose number of simultaneous matches among other sample images that have been matched simultaneously satisfies a predetermined condition (within the top predetermined number)" ("Top simultaneous detections" in the figure). Note that the second item is not limited to this example.

[0110] In Fig. 10, one sample image is selected as the selected sample image by operating the "Top simultaneous detection" button. When the "Top simultaneous detection" button is operated, "sample images whose number of simultaneous matches satisfies a predetermined condition (within the top predetermined number) among other sample images that have been matched simultaneously with the selected sample image" are output, as shown in Fig. 11.

[0111] In the example shown in Fig. 10, operating the "Distribution" button outputs "distribution information in which the feature quantities of each sample image represented by a multi-dimensional vector are reduced in dimension, and then multiple sample images are plotted on a graph of the reduced dimension," as described in the fourth embodiment. Fig. 12 shows an example of the distribution information that is output. In the example shown, the selected sample image, sample images that are likely to match simultaneously with the selected sample image, and other sample images are displayed separately.

[0112] The tally output unit 14 may also perform the highlighting described in the fourth embodiment in the rank display as shown in Fig. 10. The highlighting method is the same as that described in the fourth embodiment.

[0113] Other configurations of the processing apparatus 10 of this embodiment are similar to those of the processing apparatus 10 of the first to third embodiments.

[0114] According to the processing apparatus 10 of this embodiment, the same effects as those of the processing apparatus 10 of the first to third embodiments are achieved.

[0115] Furthermore, the processing device 10 of this embodiment can output summary information for selected sample images using a characteristic method as shown in Figures 10 to 12. Specifically, the processing device 10 can rank-display multiple sample images based on the values ​​of predetermined items indicated in the summary information. The processing device 10 can then output summary information for a selected sample image selected from the ranked sample images. Based on the output information, the user can grasp the overall picture of the multiple sample images and the details of each sample image.

[0116] <Modification> The processing device 10 may have a function for deleting a sample image designated by a user from among the sample images stored in the storage device. For example, as shown in Fig. 9, a button for deleting the selected sample image may be displayed in information (a speech bubble) associated with the selected sample image. Then, the processing device 10 may delete the selected sample image from among the sample images stored in the storage device in response to operation of the button.

[0117] Furthermore, the processing device 10 may display a button for deleting each sample image in association with each sample image in the rank display as shown in Fig. 10. Then, the processing device 10 may delete the selected sample image from the sample images stored in the storage device in response to an operation of the button.

[0118] The processing device 10 may also have a function for collectively specifying and deleting multiple sample images. For example, the processing device 10 may accept a user input specifying an area on a graph in distribution information such as those shown in Figures 8, 9, and 12. Then, in response to a subsequent deletion instruction from the user, the processing device 10 may delete one or more sample images present in the specified area from the sample images stored in the storage device.

[0119] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations may be adopted. The configurations of the above-described embodiments may be combined with each other, or some of the configurations may be replaced with other configurations. Furthermore, various modifications may be made to the configurations of the above-described embodiments without departing from the spirit of the invention. Furthermore, the configurations and processes disclosed in the above-described embodiments and modified examples may be combined with each other.

[0120] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order. However, the order of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed to the extent that the content is not affected. Furthermore, the above-described embodiments can be combined to the extent that the content is not contradictory.

[0121] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes: 1. A processing device having: an acquisition means for acquiring a search query; a search means for comparing the search query with a plurality of sample images and searching for the sample images that match the search query; a registration means for registering the results of the matching in a matching history that indicates one or more of the sample images that match in each matching; and a compilation output means for generating and outputting compilation information regarding the results of the matching for a selected sample image from the plurality of sample images based on the comparison history. 2. The processing device described in 1, wherein the search means searches for the sample images whose similarity with the search query is equal to or greater than a reference value as the sample images that match the search query, and the aggregate information regarding the selected sample images indicates at least one of the number of matches, the matching rate, statistics on the number of other sample images that have been matched simultaneously, other sample images that have been matched simultaneously, the number of times that each of the other sample images that have been matched simultaneously has been matched simultaneously, the sample images among the other sample images that have been matched simultaneously that satisfy a predetermined condition for the number of times that they have been matched simultaneously, and the similarity with the other sample images that have been matched simultaneously. The search means searches for sample images whose similarity to the search query is equal to or greater than a reference value as sample images that match the search query, and ranks the sample images that match the search query in order of highest similarity, and the aggregated information regarding the selected sample image indicates at least one of the following: a statistical value of the rank at the time of matching, a statistical value of the rank of each of other sample images that have been matched at the same time when they were matched simultaneously, a statistical value of the rank difference between the rank at the time of matching and the rank of each of other sample images that have been matched at the same time when they were matched simultaneously, and a history of the rank pairs of other sample images that have been matched at the same time and the selected sample image.4. The processing device according to any one of 1 to 3, wherein the tallying and outputting means reduces the dimension of the feature amount of each of the sample images represented by a multi-dimensional vector, then generates and outputs distribution information in which the plurality of sample images are plotted on a graph of the reduced dimension, accepts input to select the selected sample image on the graph, and outputs the tally information related to the selected selected sample image. 5. The processing device according to 4, wherein the tallying and outputting means highlights, of the sample images plotted on the graph, those sample images for which the value of a predetermined item indicated in the tally information satisfies an emphasis condition. 6. The processing device according to 5, wherein the tally outputting means sets the emphasis condition based on user input. 7. The processing device according to any one of 1 to 6, wherein the tally outputting means ranks the plurality of sample images based on the value of the predetermined item indicated in the tally information, accepts input to select the selected sample image from the plurality of sample images displayed in the ranks, and outputs the tally information related to the selected selected sample image. 8. The processing device according to claim 7, wherein the tally output means displays, in the rank display, a value of a first item indicated in the tally information linked to each of the plurality of sample images, and displays a value of a second item indicated in the tally information as the tally information related to the selected sample image. 9. A processing method in which one or more computers acquire a search query, match the search query with a plurality of sample images, search for sample images that match the search query, register the results of the matching in a matching history that indicates one or more sample images that match in each matching, and generate and output tally information related to the results of the matching for a selected sample image from the plurality of sample images based on the matching history.10. A program that causes a computer to function as: an acquisition means that acquires a search query; a search means that compares the search query with a plurality of sample images and searches for the sample images that match the search query; a registration means that registers the results of the comparison in a comparison history that indicates one or more of the sample images that match in each comparison; and a compilation output means that generates and outputs compilation information regarding the results of the comparison for a selected sample image from the plurality of sample images based on the comparison history.

[0122] This application claims priority based on Japanese Patent Application No. 2022-207926, filed December 26, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0123] 10 Processing device 11 Acquisition unit 12 Search unit 13 Registration unit 14 Aggregation output unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. An acquisition means for acquiring a search query; a search means for matching the search query with a plurality of sample images and searching for the sample images that match the search query; a registration means for registering the result of the matching in a matching history indicating one or more of the sample images matched in each matching; a compilation output means for generating compilation information on the results of the comparison for a selected sample image from among the plurality of sample images based on the comparison history and outputting the compilation information; A processing device having:

2. The search means searching for the sample images whose similarity to the search query is equal to or greater than a reference value as the sample images matching the search query; The aggregate information regarding the selected sample images is The processing device according to claim 1, which indicates at least one of the number of matches, the matching rate, a statistical value of the number of other sample images that have been matched simultaneously, other sample images that have been matched simultaneously, the number of times that each of the other sample images that have been matched simultaneously has been matched simultaneously, the sample images among the other sample images that have been matched simultaneously whose number of times that they have been matched simultaneously satisfies a predetermined condition, and the similarity with the other sample images that have been matched simultaneously.

3. The search means searching for the sample images whose similarity to the search query is equal to or greater than a reference value as the sample images that match the search query, and ranking the sample images that match the search query in descending order of similarity; The aggregate information regarding the selected sample images is A processing device as described in claim 1 or 2, which indicates at least one of the following: a statistical value of the ranking at the time of matching; a statistical value of the ranking of each of the other sample images that have been matched at the same time when they were matched simultaneously; a statistical value of the difference in ranking between the ranking at the time of matching and the ranking of each of the other sample images that have been matched at the same time when they were matched simultaneously; and a history of pairs of the rankings of the selected sample image and other sample images that have been matched at the same time.

4. The tally output means After reducing the dimension of the feature amount of each of the sample images represented by a multi-dimensional vector, generating and outputting distribution information in which the plurality of sample images are plotted on a graph of the reduced dimension, Accepting an input to select the selected sample image on the graph; The processing device according to claim 1 or 2, wherein the collected information relating to the selected sample image is output.

5. The tally output means 5. The processing device according to claim 4, wherein, of the sample images plotted on the graph, sample images whose values ​​of predetermined items indicated by the summary information satisfy a highlighting condition are highlighted.

6. The tally output means The processing device according to claim 5 , wherein the emphasis conditions are set based on a user input.

7. The tally output means displaying the plurality of sample images in rank order based on the values ​​of the predetermined items indicated by the aggregate information; Accepting an input to select the selected sample image from the plurality of sample images displayed in the rank order; The processing device according to claim 1 or 2, wherein the collected information relating to the selected sample image is output.

8. The tally output means In the rank display, a value of a first item indicated by the aggregate information is displayed in association with each of the plurality of sample images; The processing device according to claim 7 , wherein the value of a second item indicated in the summary information is displayed as the summary information regarding the selected sample image.

9. One or more computers Get the search query, Matching the search query with a plurality of sample images to search for sample images that match the search query; registering the results of the matching in a matching history indicating one or more of the sample images matched in each matching; A processing method for generating and outputting summary information on the results of the matching for a selected sample image from among the plurality of sample images based on the matching history.

10. Computer, A means for obtaining a search query; a search means for matching the search query with a plurality of sample images and searching for the sample images that match the search query; a registration means for registering the result of the matching in a matching history indicating one or more of the sample images matched in each matching; a compilation output means for generating compilation information on the results of the comparison for a selected sample image from among the plurality of sample images based on the comparison history and outputting the compilation information; A program that functions as a