Image selection device, image selection method, and computer program

Abstract

To enable obtaining various types of images as selection results from among a plurality of images. [Solution] An image selection device comprising: a diversity information acquisition unit that acquires diversity information representing the diversity of a provisionally selected image group when a candidate image, which is an image included in a target image group and is being judged, is added to a provisionally selected image group that includes one or more images selected as a provisional image group from a target image group that includes multiple images to be processed; and an image selection unit that acquires the diversity information for multiple candidate images included in the target image group, selects new candidate images to add to the provisionally selected image group based on the diversity information, and determines the provisionally selected image group as a selected image group when a specified termination condition is met; and if the specified termination condition is not met, the diversity information acquisition unit acquires the diversity information using a new virtual selected image group to which the candidate image selected by the image selection unit has been added.

Description

[Technical Field]

[0001] The present invention relates to an image selection device, an image selection method, and a computer program. [Background technology]

[0002] Conventionally, mechanisms for searching for a desired image from among a large number of images have been proposed. For example, there is a technology that acquires features for each image to be searched using a predetermined algorithm, searches for images corresponding to a query provided by a user, and presents them to the user. In recent years, a huge amount of image data has been generated, and the need for technology to search for such images has increased. For example, the technology disclosed in Patent Document 1 operates as follows: First, a reliability value of tag information related to a keyword (query) is acquired, and area information of a subject related to the keyword is acquired from the image to be searched. The proportion of the subject that occupies the entire image is calculated, and an evaluation value is calculated using the reliability value of the tag information and the proportion of the subject that occupies the entire image. Then, image search results that reflect the evaluation value are returned. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-104364 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, when multiple images related to a user's query are retrieved as search results, many similar images are returned as top search results, resulting in the display of only similar images. Therefore, if the image the user expected is not displayed as a search result, the user must repeatedly refresh the display until the desired image is displayed.

[0005] The present invention has been made in view of the above circumstances, and provides a technique that makes it possible to obtain various types of images as selection results from among a plurality of images. [Means for solving the problem]

[0006] One aspect of the present invention is an image selection device that includes: a diversity information acquisition unit that acquires diversity information representing the diversity of a provisionally selected image group when a candidate image, which is an image included in a target image group and is being judged, is added to a provisionally selected image group that includes one or more images selected as a provisional image group from a target image group that includes a plurality of images to be processed; and an image selection unit that acquires the diversity information for a plurality of candidate images included in the target image group, selects a new candidate image to be added to the provisionally selected image group based on the diversity information, and determines the provisionally selected image group as a selected image group when a predetermined termination condition is satisfied; and if the predetermined termination condition is not satisfied, the diversity information acquisition unit acquires the diversity information using a new virtual selected image group to which the candidate image selected by the image selection unit has been added.

[0007] One aspect of the present invention is the above-mentioned image selection device, which further includes a relevance information acquisition unit that acquires relevance information indicating the relevance between a pre-specified query and the candidate images, and the image selection unit selects new candidate images to add to the group of provisionally selected images based on the diversity information and the relevance information.

[0008] One aspect of the present invention is the above-mentioned image selection device, wherein the image selection unit performs processing based on information obtained based on a new provisionally selected image group that includes images selected from the images included in the selected image group selected by the image selection unit, and determines the new provisionally selected image group as the new selected image group at the time a predetermined termination condition is met.

[0009] One aspect of the present invention is an image selection method comprising: a diversity information acquisition step of acquiring diversity information representing the diversity of a provisionally selected image group when a provisionally selected image group containing one or more images selected as a provisional image group from a target image group containing a plurality of images to be processed is added with a candidate image, which is an image included in the target image group and is being judged; an image selection step of acquiring the diversity information for a plurality of candidate images included in the target image group and selecting a new candidate image to be added to the provisionally selected image group based on the diversity information; and a determination step of determining the provisionally selected image group as a selected image group when a predetermined termination condition is satisfied; and if the predetermined termination condition is not satisfied, the diversity information acquisition step is executed using a new virtual selected image group to which the candidate image selected in the image selection step has been added.

[0010] One aspect of the present invention is a computer program that causes a computer to function as an image selection device, comprising: a diversity information acquisition unit that acquires diversity information representing the diversity of a provisionally selected image group when a provisionally selected image group containing one or more images selected as a provisional image group from a target image group containing a plurality of images to be processed is added with a candidate image that is an image included in the target image group and is being judged; and an image selection unit that acquires the diversity information for a plurality of candidate images included in the target image group, selects a new candidate image to be added to the provisionally selected image group based on the diversity information, and determines the provisionally selected image group as a selected image group when a predetermined termination condition is met; and if the predetermined termination condition is not met, the diversity information acquisition unit acquires the diversity information using a new virtual selected image group to which the candidate image selected by the image selection unit has been added. [Effects of the Invention]

[0011] The present invention makes it possible to obtain various types of images as selection results from among a plurality of images. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic block diagram showing the system configuration of an image selection system 100 according to the present invention. [Figure 2] 2 is a schematic block diagram showing a specific example of the functional configuration of the terminal device 10. FIG. [Figure 3] 2 is a schematic block diagram showing a specific example of the functional configuration of the image selection device 20. FIG. [Figure 4] 10 is a flowchart showing a specific example of the flow of the selected image group determination process of the image selection device 20. [Figure 5] FIG. 10 is a diagram showing a specific example of a selected image group screen. [Figure 6] FIG. 10 is a diagram showing an example of a display arranged according to a predetermined standard. [Figure 7] 10 is a flowchart showing a specific example of the flow of an image group presentation process, which is one application example of the selected image group determination process. [Figure 8] FIG. 10 is a diagram showing a specific example of a selected image group screen displayed in the image group presentation process. [Figure 9] FIG. 10 is a diagram showing a specific example of a selected image group screen displayed in the image group presentation process. [Figure 10] FIG. 10 is a diagram showing a specific example of a selected image group screen displayed in the image group presentation process. [Figure 11] FIG. 2 is a diagram illustrating an outline of an example of the hardware configuration of an information processing device 90 applied to the present embodiment. [Figure 12] FIG. 10 is a diagram showing a modified example of the image selection device 20. DETAILED DESCRIPTION OF THE INVENTION

[0013] FIG. 1 is a schematic block diagram showing the system configuration of an image selection system 100 of the present invention. The image selection system 100 selects various types of images from a plurality of images to be searched (hereinafter referred to as a "target image group") so as to obtain diversity in image content. The images included in the target image group may be still image data or moving image data. In the following explanation, a case will be described in which the images included in the target image group are configured using still image data.

[0014] The image selection system 100 includes a terminal device 10 and an image selection device 20. The terminal device 10 and the image selection device 20 are communicably connected via a network 70. The network 70 may be a network using wireless communication or a network using wired communication. The network 70 may be configured using, for example, the Internet or a local area network (LAN). The network 70 may also be configured by combining a plurality of networks.

[0015] 2 is a schematic block diagram showing a specific example of the functional configuration of the terminal device 10. The terminal device 10 is configured using information equipment such as a smartphone, tablet, personal computer, dedicated device, etc. The terminal device 10 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15.

[0016] The communication unit 11 is a communication device. The communication unit 11 may be configured as, for example, a network interface. The communication unit 11 communicates data with other devices via the network 70 in accordance with the control of the control unit 15. The communication unit 11 may be a device that performs wireless communication or a device that performs wired communication.

[0017] The input unit 12 is configured using existing input devices such as a keyboard, a pointing device (mouse, tablet, etc.), buttons, a touch panel, etc. The input unit 12 is operated by a user when inputting user instructions to the terminal device 10. The input unit 12 may be an interface for connecting the input device to the terminal device 10. In this case, the input unit 12 inputs an input signal generated in the input device in response to a user input to the terminal device 10. The input unit 12 may be configured using a microphone and a voice recognition device. In this case, the input unit 12 performs voice recognition on words spoken by the user and inputs character string information of the recognition result to the terminal device 10. In this case, the voice recognition may be performed by the control unit 15. The input unit 12 may be configured in any way as long as it is configured to be able to input user instructions to the terminal device 10.

[0018] The input unit 12 may be used, for example, for a user to input information indicating a query. The query may be, for example, a keyword (text) designated by the user, or an image designated by the user. One query or multiple queries may be designated. When multiple queries are designated, the designated multiple queries are transmitted to the image selection device 20. The keyword and image may be input by the user, or may be selected by the user from among candidates. When an image is input, a pre-generated image may be input to the terminal device 10 from another recording medium or another information processing device, or an image may be input to the terminal device 10 by capturing an image using an imaging device included in the terminal device 10. The input unit 12 may be used, for example, to select a predetermined number of images from a group of selected images displayed on the output unit 13.

[0019] The output unit 13 outputs information in a form that can be recognized by the user. The output unit 13 may be, for example, an image display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The output unit 13 may also be an interface for connecting an image display device to the terminal device 10. In this case, the output unit 13 generates a video signal for displaying image data and outputs the video signal to the image display device connected to the output unit 13. The output unit 13 may also be configured as a touch panel integrated with the input unit 12. The output unit 13 may also display, for example, a group of selected images selected by the image selection device 20.

[0020] The storage unit 14 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 14 stores data used by the control unit 15. The storage unit 14 stores data required when the control unit 15 performs processing.

[0021] The control unit 15 is configured using a processor such as a CPU (Central Processing Unit) and a memory (main storage device). The control unit 15 functions when the processor executes a program. Note that all or part of the functions of the control unit 15 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., SSDs: Solid State Drives), as well as storage devices such as hard disks and semiconductor storage devices built into computer systems. The program may be transmitted via a telecommunications line.

[0022] The control unit 15 may execute, for example, an application installed on its own device (terminal device 10). A specific example of such an application is an application provided to the terminal device 10 as a dedicated application for the image selection system 100. Another specific example of such an application is a web browser application. Such an application may be pre-installed on the terminal device 10, or may be downloaded each time an image selection process is performed. For example, when implemented as a web browser application, the terminal device 10 may download and execute the application from a device specified by a specific web server (for example, the web server itself or another server) in response to the terminal device 10 connecting to the web server. The control unit 15 operates according to the program of the application being executed.

[0023] The control unit 15 controls the terminal device 10 in response to user operations and information received from the image selection device 20. For example, the control unit 15 transmits information input by the user operating the input unit 12 to the image selection device 20 by using the communication unit 11. Specific examples of information input to the terminal device 10 include information indicating a query and information indicating an image selected from a group of selected images.

[0024] For example, when information transmitted from image selection device 20 is received by communication unit 11 via network 70, control unit 15 generates screen data based on the received information and causes output unit 13 to display the screen data. Such screen data includes images and text indicating the information transmitted from image selection device 20. For example, when information transmitted from image selection device 20 is received by communication unit 11 via network 70, control unit 15 may generate audio data based on the received information and cause output unit 13 to output the audio data.

[0025] 3 is a schematic block diagram showing a specific example of the functional configuration of the image selection device 20. The image selection device 20 is configured using an information processing device such as a personal computer or a server device. The image selection device 20 includes a communication unit 21, a storage unit 22, and a control unit 23.

[0026] The communication unit 21 is a communication device. The communication unit 21 may be configured as, for example, a network interface. The communication unit 21 communicates data with other devices via the network 70 in accordance with the control of the control unit 23. The communication unit 21 may be a device that performs wireless communication or a device that performs wired communication.

[0027] The storage unit 22 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 22 stores data used by the control unit 23. The storage unit 22 may function as, for example, a target image group storage unit 221, an image feature amount storage unit 222, a relevance information storage unit 223, and a diversity information storage unit 224.

[0028] The target image group storage unit 221 stores a group of images (plurality of images) to be searched. The target image group may be stored in advance in the image selection device 20, or may be transmitted from the terminal device 10. The target image group may be acquired from another information processing device (for example, an image database). The target image group may also be obtained as a search result from a search engine on the Internet.

[0029] The image feature amount storage unit 222 stores image feature amounts acquired by the control unit 23 (image feature amount acquisition unit 232). The image feature amount storage unit 222 stores, for example, image feature amounts related to one or more (or all) images included in the target image group. When storing multiple image feature amounts, the image feature amount storage unit 222 stores each image in association with the image feature amount. The image feature amount may be acquired using any algorithm. For example, the image feature amount may be a value that changes depending on the color of the entire image, a value that changes depending on the attributes of an object included in the image (e.g., the type or size of the object), or a value that changes depending on the meaning of characters or character strings included in the image. The image feature amounts related to each image included in the target image group are acquired using the same method (the same algorithm).

[0030] The relevance information storage unit 223 stores the relevance information acquired by the control unit 23 (relevance information acquisition unit 233). The relevance information is information indicating the relevance of each image included in the target image group to the content of a specified query. The query may be designated by the user, designated by another information processing device, or randomly generated. When storing multiple pieces of relevance information, the relevance information storage unit 223 stores each image in association with the relevance information. The relevance information may be information indicating, for example, the level of relevance. More specifically, the relevance information may be represented using the distance between the feature amount of the query and the image feature amount of the image being processed (the image being the target of relevance information acquisition). In other words, the relevance information may be represented using the distance between the vector indicating the feature amount of the query and the vector indicating the image feature amount of the image being processed. In this case, the smaller the value of the relevance information, the higher the relevance between the query and the image. In this case, the query feature amount may be acquired using any algorithm. However, it is preferable that the query feature amount be acquired using the same method (same algorithm) as the image feature amount. In this regard, since the query may be expressed as text data as described above, a multimodal algorithm may be used to acquire the feature amount. Note that the inverse of the distance described above may be used as the relevance information.

[0031] The diversity information storage unit 224 stores diversity information acquired by the control unit 23 (diversity information acquisition unit 234). Diversity information indicates the degree of diversity of images included in a target image group (a provisionally selected image group in this embodiment). For example, diversity information is information indicating the degree of variation in image feature amounts of images included in the target image group. For example, high diversity indicates that the image group includes a variety of types of images (images with different styles), and low diversity indicates that the image group includes many similar types of images (images with similar styles). When storing multiple pieces of diversity information, the diversity information storage unit 224 stores each image and its corresponding diversity information in association with each other. Any algorithm may be used to obtain the diversity information, as long as it is an algorithm that can obtain information indicating the diversity described above.

[0032] In this embodiment, the diversity information indicates the diversity of the temporarily selected image group when each image included in the target image group is added to the temporarily selected image group currently selected. The temporarily selected image group is an image group including one or more images selected as a temporary image group. All images included in the temporarily selected image group are images selected from the target image group. For example, diversity information may be indicated using the volume value of a parallelepiped in N dimensions (parallel volume) formed by the image feature quantities (vectors) of all images included in the temporarily selected image group. The more similar vectors are included, the smaller the volume of the parallelepiped, and the more orthogonal vectors are included, the larger the volume of the parallelepiped. Therefore, diversity can be appropriately indicated using the volume of such a parallelepiped. More specifically, diversity may be indicated as V in Equation 1 below. In this case, the volume V may be expressed as the square root of the determinant det(G) of the Gram matrix G of the vectors (image feature quantities) of N images included in the temporarily selected image group.

[0033]

number

[0034]

number

[0035] As the diversity information, the above-mentioned volume V may be applied as is, or det(G) may be used, or the logarithm of det(G) (log(det(G))) may be applied. Alternatively, the sum of the distances of the image feature amounts (vectors) between each image in the temporarily selected image group and the candidate image or the DPP (Determinantal Point Process) may be used as the diversity information.

[0036] The control unit 23 is configured using a processor such as a CPU and a memory. The control unit 23 functions as an information control unit 231, an image feature acquisition unit 232, a relevance information acquisition unit 233, a diversity information acquisition unit 234, and an image selection unit 235 by the processor executing a program. Note that all or part of the functions of the control unit 23 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., an SSD), and storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0037] The information control unit 231 acquires information indicating a query and information regarding a group of target images to be processed from another device such as the terminal device 10. The information control unit 231 transmits information indicating the group of selected images selected by the image selection unit 235 to the other device such as the terminal device 10. Such exchange of information between the information control unit 231 and another device may be performed by communication using the communication unit 21, for example.

[0038] The image feature amount acquiring unit 232 acquires the image feature amount of each image included in the target image group, and records the acquired image feature amount in the image feature amount storage unit 222.

[0039] The relevance information acquisition unit 233 acquires relevance information with respect to the query for each image included in the target image group, and records the acquired relevance information in the relevance information storage unit 223.

[0040] The diversity information acquisition unit 234 acquires, for each image included in the target image group, diversity information of the temporarily selected image group when that image is included in the temporarily selected image group. The diversity information acquisition unit 234 records the acquired diversity information in the diversity information storage unit 224.

[0041] The image selection unit 235 selects images to add to the provisionally selected image group from among the images included in the target image group. Such images are selected from among images that have not yet been selected in the provisionally selected image group. Furthermore, when a condition for terminating the selection process is satisfied, the image selection unit 235 determines the multiple images currently selected in the provisionally selected image group as the selected image group. Information indicating the selected image group determined in this manner is transmitted by the information control unit 231 to other devices such as the terminal device 10.

[0042] FIG. 4 is a flowchart showing a specific example of the flow of the selected image group determination process of the image selection device 20. The selected image group determination process is a process for selecting a wide variety of images from a plurality of images, thereby achieving a highly diverse selection of images. Note that the flowchart in FIG. 4 does not describe the process for acquiring a query selected by the user, but it is assumed that the query has been acquired in advance before the flowchart starts. For example, when the user inputs a query into the terminal device 10 by operating the input unit 12 of the terminal device 10, the terminal device 10 transmits the query to the image selection device 20. The information control unit 231 of the image selection device 20 acquires the query by receiving it from the terminal device 10.

[0043] First, the image selection unit 235 selects an initial image and adds it to a group of temporarily selected images (step S101). The initial image may be selected under any selection conditions. For example, the image selection unit 235 may randomly select an initial image from a group of target images. For example, the image selection unit 235 may select, as the initial image, an image that has the highest degree of relevance indicated by the relevance information among the images included in the group of target images. The image selection unit 235 may select, as the initial image, an image specified by the user. Such a selection by the user may be made, for example, by selecting from a group of target images, or by selecting from a group of selected images selected by processing of the image selection device 20.

[0044] Next, the image selection unit 235 selects a candidate image from among the images included in the target image group that are not included in the provisionally selected image group (step S102). Next, the image feature amount acquisition unit 232 acquires the image feature amount of the candidate image (step S103). If the image feature amount of the candidate image has not yet been calculated, the image feature amount acquisition unit 232 calculates the image feature amount and records it in the image feature amount storage unit 222. If the image feature amount of the candidate image has already been stored in the image feature amount storage unit 222, the image feature amount acquisition unit 232 may acquire the image feature amount by reading it out from the image feature amount storage unit 222.

[0045] Next, the relevance information acquisition unit 233 acquires relevance information between the query and the image feature amount of the candidate image (step S104). If the relevance information between the query and the candidate image has not yet been calculated, the relevance information acquisition unit 233 calculates the relevance information and records it in the relevance information storage unit 223. If the relevance information of the candidate image has already been stored in the relevance information storage unit 223, the relevance information acquisition unit 233 may acquire the relevance information by reading it out from the relevance information storage unit 223.

[0046] Next, diversity information acquisition unit 234 acquires diversity information of the temporarily selected image group when the candidate image is added to the temporarily selected image group by calculating it (step S105). After calculating the diversity information, diversity information acquisition unit 234 records it in diversity information storage unit 224. If processing as a candidate image (acquisition of relevance information and diversity information) has not been completed for all images included in the target image group (step S106-NO), image selection unit 235 selects a new candidate image (step S102). In this case, the processing of steps S103 to S105 is executed for the newly selected candidate image.

[0047] If the processing as a candidate image (acquisition of relevance information and diversity information) has been completed for all images included in the target image group (step S106-YES), the image selection unit 235 selects candidate images that satisfy the selection conditions as images to be added to the provisionally selected image group (step S107). The selection conditions may be, for example, conditions related to the relevance information and diversity information. Such selection conditions may be defined, for example, as conditions related to the score obtained based on the relevance information and diversity information. Specifically, if the score is a value that increases according to the level of relevance and the level of diversity, the selection condition may be the image with the largest score. The score may be calculated, for example, based on the following equations 3 and 4. In the following equations, r represents the value of the relevance information (for example, the vector distance).

[0048]

number

[0049]

number

[0050] The selection conditions may include, for example, a threshold value related to diversity information (hereinafter referred to as the "diversity threshold"), and images to be added to the group of temporarily selected images may be selected from candidate images related to diversity information that satisfies the condition related to the diversity threshold. Specifically, a diversity threshold indicating that diversity is higher than a certain value may be set, and images to be added to the group of temporarily selected images may be selected from candidate images related to diversity information higher than this diversity threshold. The selection conditions may be determined according to the number of images included in the group of temporarily selected images. In other words, the selection conditions may change according to the number of images included in the group of temporarily selected images. Note that, in one processing of step S107, one candidate image may be selected and added to the group of temporarily selected images, or multiple candidate images may be selected and added to the group of temporarily selected images.

[0051] Next, the image selection unit 235 determines whether a termination condition is satisfied (step S108). The termination condition may be defined, for example, as the number of images included in the temporarily selected image group being equal to or greater than a predetermined threshold, as the number of images in the target image group that are not included in the temporarily selected image group being less than a predetermined threshold, as the maximum score among the scores of the candidate images being less than a predetermined threshold, or as some other condition. If the termination condition is not satisfied (step S108—NO), the process returns to step S102, and the processes from step S102 onward are executed. If the termination condition is satisfied (step S108—YES), the image selection unit 235 determines the temporarily selected image group at that time as the selected image group (step S109).

[0052] The information indicating the selected image group obtained by the processing of step S109 may be used in any manner. For example, the information indicating the selected image group may be transmitted to terminal device 10, and a screen for displaying each image included in the selected image group (hereinafter referred to as a "selected image group screen") may be displayed on terminal device 10. If data for each image is already stored in terminal device 10, only identification information for the selected images (images included in the selected image group) may be transmitted. If data for each image is not stored in terminal device 10, data for each selected image may be transmitted. Terminal device 10 displays the selected image group screen on output unit 13.

[0053] FIG. 5 is a diagram showing a specific example of a selected image group screen. The selected image group screen is provided with a plurality of image display areas 31. Each image selected as part of the selected image group is displayed in each image display area 31. At this time, each image may be displayed reduced while maintaining its aspect ratio so that it fits within the image display area 31. On the selected image group screen, images may be displayed in an arrangement according to a predetermined standard. FIG. 6 is a diagram showing an example of a display arranged according to a predetermined standard. For example, images may be displayed following a predetermined image (e.g., an initial image) in order of increasing similarity to the initial image, or images may be displayed in an arrangement in a predetermined orientation in order of increasing relevance information value (high relevance) to the query.

[0054] For example, the images may be displayed in a predetermined orientation in order of the diversity information value. Specifically, for example, the process is as follows: The images displayed on the selected image group screen (images selected as the selected image group) are set as a new target image group, and a new initial image is selected from the new target image group. The new initial image may be, for example, the initial image selected first in the selected image group (the initial image selected in the processing of step S101), an image selected by the user, or an image selected using other logic. The new initial image is stored in association with the order of selection (e.g., "1"). Next, the new initial image in this new target image group is added to a new temporarily selected image group, and diversity information for all images in the new target image group when added to the new temporarily selected image group is calculated. Among these, the image with the lowest diversity information value (lowest diversity) is added to the new temporarily selected image group and stored in association with the selection order. This process is performed until all images in the new target image group have been selected, and an order is assigned to all images. Subsequently, the images are sorted in the associated order and displayed, thereby realizing a display in which images are arranged in order from most similar to the new initial image.

[0055] By displaying similar images in this way, it is possible to improve visibility because similar images are displayed together. The nine images shown in Figure 5 were displayed separately, but in Figure 6, related images are displayed together. For example, the top row contains images of lipstick itself, the middle row to the bottom left contains images of women's faces and close-ups of their lips, and the bottom center to right contains other images.

[0056] Another example of the use of the selected image group is, for example, when a user selects multiple images from multiple pieces of image data taken by the user to create an album, the selected image group can be used to select multiple images to be used in the album.

[0057] FIG. 7 is a flowchart showing a specific example of the flow of an image group presentation process, which is one application example of the selected image group determination process. The image group presentation process is a process in which the selected image group determination process is repeated multiple times to present the user with an image group that more closely matches the user's request. First, a query selected by the user is acquired (step S201). Next, the selected image group determination process is executed based on the acquired query (step S202). Next, the information control unit 231 instructs the terminal device 10 to display a screen (selected image group screen) for displaying each image included in the selected image group (step S203). At this time, the information control unit 231 transmits information indicating the images included in the selected image group to the terminal device 10. If the data of each image is already stored in the terminal device 10, only identification information of the selected image may be transmitted. If the data of each image is not stored in the terminal device 10, the data of each selected image may be transmitted. The terminal device 10 displays the selected image group screen on the output unit 13.

[0058] The information control unit 231 determines whether or not a termination condition has been met (step S204). The termination condition may be, for example, a user indicating that no more selected images need to be selected. The termination condition may be, for example, a selected image group determination process having been repeatedly executed a predetermined number of times. If the termination condition has been met (step S204-YES), the processing of FIG. 7 ends.

[0059] If the termination condition is not satisfied (step S204—NO), the process waits until the user selects an image on the selected image group screen. When the user selects an image (step S205—YES), the terminal device 10 transmits information indicating the selected image to the image selection device 20. The image selection unit 235 of the image selection device 20 determines the selected image as an initial image. Furthermore, the image selection unit 235 determines a diversity threshold according to the number of repeated selections (step S206). The number of repeated selections is the number of times the process of step S202 (selected image group determination process) is repeatedly executed in the flowchart of FIG. 7. The greater the number of repeated selections, the lower the diversity threshold may be determined to be. For example, linear decay or exponential decay may be applied to the process of lowering the diversity threshold. Then, the selected image group determination process is executed using the determined initial image and diversity threshold. By performing this process in this manner, the greater the number of repeated selections, the more candidate images with low diversity information values ​​will be selected if their relevance information is high, and the relevance information becomes more dominant than the diversity information. Therefore, when the number of repeated selections is small, a more diverse group of selected images is displayed, and as the number of repeated selections increases, more highly related and similar images are displayed in the group of selected images.

[0060] In the process of step S206, the diversity threshold may be set so that the upper limit of the diversity information is lower. By configuring in this way, candidate images with too much diversity will not be selected. Therefore, the more repeated selections are made, the more similar images will be displayed in the selected image group.

[0061] FIG. 8 shows a specific example of a selected image group screen displayed during the image group presentation process. In FIG. 8, the number of steps and the selected search query are displayed. The number of steps is displayed as "1 / 6" in FIG. 8. In this case, the "1" in the upper left indicates the number of times the selected image group determination process has been executed, and the "6" in the lower right indicates the maximum number of times the selected image group determination process has been repeated. Reaching the maximum number of times the process has been executed is one of the conditions for terminating step S204 in FIG. 7. The search query indicates a query specified by the user. In the example of FIG. 8, three character strings, "red," "lipstick," and "makeup," were specified as the query. In the example of FIG. 8, nine images related to this query are displayed as the selected image group. Because the number of repetitions is one, images are selected to have a relatively high level of diversity. Therefore, the images displayed include an image of lipstick itself, an image of a woman's face with lipstick applied, an image of a clown's face with lipstick applied, an image of a woman's upper body with lipstick applied, and an image of the lips with lipstick applied, and each image of the woman's face has a different expression.

[0062] On the selected image group screen displayed in the image group presentation process, an image selection button 32 and a decision button 33 are displayed in association with each image display area 31. The image selection button 32 is a button for indicating that the image is one that the user tends to want to select, and that a new selected image group screen will be displayed by executing the selected image group determination process. Operating the image selection button 32 corresponds to the selection in step S205 in FIG. 7. When the image selection button 32 is operated, the image corresponding to the operated image selection button 32 is set as the initial image, and a diversity threshold is determined according to the number of repeated selections. Then, the selected image group determination process is executed, and a new selected image group screen is displayed.

[0063] The decision button 33 is a button for indicating that the image is the image that the user ultimately wants to select. Operating the decision button 33 is one of the termination conditions. Therefore, when the decision button 33 is operated, the selected image group determination process is not repeated any further, and the image group presentation process ends.

[0064] 9 is a diagram showing a specific example of a selected image group screen displayed in the image group presentation process. FIG. 9 shows a specific example of a selected image group screen displayed when the image selection button 32 is operated for the face image of the woman in the center of the top row in FIG. 8. Therefore, FIG. 9 shows the results of executing the selected image group determination process (step S202) with the face image of the woman in the center of the top row in FIG. 8 selected as the initial image. In FIG. 9, the diversity threshold has been changed, so the overall diversity is lower than in FIG. 8. For example, the face images of the three women in the top row all have expressions with their mouths closed, and the lipstick colors are similar. Furthermore, the images of the lipstick themselves also have a similar overall color.

[0065] FIG. 10 is a diagram showing a specific example of a selected image group screen displayed in the image group presentation process. FIG. 10 shows a specific example of a selected image group screen displayed when the image selection button 32 is operated for the female face image on the upper left side of FIG. 9. Therefore, FIG. 10 shows the results of executing the selected image group determination process (step S202) with the female face image on the upper right side of FIG. 9 selected as the initial image. In FIG. 10, the diversity threshold has been further changed, so the overall diversity is further reduced compared to FIG. 9. For example, there are no images of lipstick itself, and all are female face images. In addition, all of the images have facial expressions with their mouths mostly closed, and the lipstick color and overall image tone are similar. In this way, the images are gradually narrowed down to those that are closer to the image selected by the user. This allows the user to more easily find the image they are looking for.

[0066] Such processing may be performed, for example, as follows. In the above description, when a user selects an image, the selected image is used as an initial image to update the diversity threshold and execute the next selected image group determination process. In contrast, the next selected image group determination process may be performed using the selected image as an additional query. By performing processing in this manner, the next selected image group determination process takes into account relevance information with the image selected by the user, and therefore an image highly relevant to the image selected by the user is selected. By repeatedly performing such processing, it is possible to achieve screen transitions similar to the specific examples of screen transitions shown in FIGS. 8 to 10.

[0067] FIG. 11 is a diagram illustrating an outline of an example hardware configuration of an information processing device 90 applied to this embodiment. The information processing device 90 includes a processor 91, a main memory device 92, a communication interface 93, an auxiliary memory device 94, an input / output interface 95, and an internal bus 96. The processor 91, the main memory device 92, the communication interface 93, the auxiliary memory device 94, and the input / output interface 95 are communicably connected to each other via the internal bus 96. The information processing device 90 may be applied to, for example, the terminal device 10 and the image selection device 20. In this case, for example, the communication units 11 and 21 may be configured using the communication interface 93. For example, the memory units 14 and 22 may be configured using the auxiliary memory device 94. Furthermore, the control units 15 and 23 may be configured using the processor 91 and the main memory device 92.

[0068] The image selection device 20 configured in this manner makes it possible to obtain various types of images as search results from among a plurality of images. Specifically, the image selection device 20 selects images based on diversity information in addition to relevance information between the query and images specified by the user. As a result, a group of images with overall diversity is selected.

[0069] (Variation) The process in FIG. 4 described above is premised on the assumption that a query has been acquired in advance. However, the selected image group determination process may be executed without using a query. In this case, for example, the relevance information value of each candidate image may be defined as a common value (e.g., zero). This configuration makes it possible to select diverse images from a target image group even in a situation where a query cannot be obtained.

[0070] When an image is input to the terminal device 10 as a query, it is transmitted as image data to the image selection device 30, and image features may be acquired by the image selection device 30, or the image features of the image input as a query may be acquired by the terminal device 10. In the latter case, the terminal device 10 may transmit the acquired image features to the image selection device 30 as image features of the query.

[0071] In this embodiment, the terminal device 10 and the image selection device 20 are configured as separate devices, but they may also be configured as an integrated device. Fig. 12 is a diagram showing a modified example of the image selection device 20 configured in this manner. The image selection device 20 shown in Fig. 12 further includes an input unit 24 and an output unit 25. The input unit 24 and the output unit 25 of the image selection device 20 shown in Fig. 12 function in the same manner as the input unit 12 and the output unit 13 of the terminal device 10, respectively. The control unit 23 operates in response to an operation on the input unit 24, performs a selected image group determination process and an image group presentation process, and outputs each screen using the output unit 25.

[0072] The image selection device 20 may be implemented using a plurality of information processing devices. For example, the image selection device 20 may be implemented using a device such as a cloud. For example, in the image selection device 20, the storage unit 22 and the control unit 23 may be implemented in different information processing devices. For example, the storage unit 22 of the image selection device 20 may be distributed and implemented in a plurality of information processing devices.

[0073] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0074] 100...image selection system, 10...terminal device, 11...communication unit, 12...input unit, 13...output unit, 14...storage unit, 15...control unit, 20...image selection device, 21...communication unit, 22...storage unit, 221...target image group storage unit, 222...image feature storage unit, 223...relevance information storage unit, 224...diversity information storage unit, 23...control unit, 231...information control unit, 232...image feature acquisition unit, 233...relevance information acquisition unit, 234...diversity information acquisition unit, 235...image selection unit, 24...input unit, 25...output unit, 31...image display area, 32...image selection button, 33...decision button, 70...network, 90...information processing device, 91...processor, 92...main storage unit, 93...communication interface, 94...auxiliary storage unit 95...input / output interface, 96...internal bus

Claims

1. a diversity information acquisition unit that acquires diversity information representing the diversity of a provisionally selected image group when a candidate image, which is an image included in the target image group and is to be determined, is added to a provisionally selected image group that includes one or more images selected as a provisional image group from a target image group that includes multiple images to be processed; an image selection unit that acquires the diversity information for a plurality of candidate images included in the target image group, selects candidate images to be newly added to the temporarily selected image group based on the diversity information, and determines the temporarily selected image group as a selected image group when a predetermined termination condition is satisfied; An image selection device, wherein if the specified termination condition is not satisfied, the diversity information acquisition unit acquires the diversity information using a new group of virtual selected images to which the candidate image selected by the image selection unit is added.

2. a relevance information acquisition unit that acquires relevance information indicating a relevance between a pre-specified query and the candidate image; The image selection device according to claim 1 , wherein the image selection unit selects a candidate image to be newly added to the group of provisionally selected images based on the diversity information and the relevance information.

3. 3. The image selection device according to claim 1, wherein the image selection unit performs processing based on information obtained based on a new provisionally selected image group that includes images selected from the images included in the selected image group selected by the image selection unit, and determines the new provisionally selected image group as the new selected image group when a predetermined termination condition is satisfied.

4. a diversity information acquisition step of acquiring diversity information representing the diversity of a provisionally selected image group when a candidate image, which is an image included in the target image group and is to be determined, is added to a provisionally selected image group including one or more images selected as a provisional image group from a target image group including a plurality of images to be processed; an image selection step of acquiring the diversity information for a plurality of candidate images included in the target image group, and selecting a candidate image to be newly added to the temporarily selected image group based on the diversity information; a determination step of determining the temporarily selected image group as a selected image group when a predetermined termination condition is satisfied, An image selection method, wherein if the predetermined termination condition is not satisfied, the diversity information acquisition step is performed using a new virtual selected image group to which the candidate image selected in the image selection step is added.

5. a diversity information acquisition unit that acquires diversity information representing the diversity of a provisionally selected image group when a candidate image, which is an image included in the target image group and is to be determined, is added to a provisionally selected image group that includes one or more images selected as a provisional image group from a target image group that includes multiple images to be processed; an image selection unit that acquires the diversity information for a plurality of candidate images included in the target image group, selects candidate images to be newly added to the temporarily selected image group based on the diversity information, and determines the temporarily selected image group as a selected image group when a predetermined termination condition is satisfied; A computer program for causing a computer to function as an image selection device in which, if the specified termination condition is not satisfied, the diversity information acquisition unit acquires the diversity information using a new group of virtual selected images to which the candidate image selected by the image selection unit is added.

Images