Information processing device, its control method, and program

Abstract

When selecting images from a group of images related to an event, it is not possible to efficiently extract images that include the main people involved in that event. [Solution] The information processing device of the present invention includes: an acquisition means for acquiring information about an object for identifying a main person based on the type of event of a group of images including multiple images; a determination means for determining the main person from the images in the group of images that contain the object based on the information about the object acquired by the acquisition means; and an extraction means for extracting images from the group of images that contain the person determined by the determination means.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method thereof, and a program.

Background Art

[0002] There is an information processing apparatus for managing images captured by electronic devices such as digital cameras and smartphones. For example, a user transmits a captured image from the electronic device to an information processing apparatus on a network and saves it. Then, the information processing apparatus transmits the image to a social network service (SNS). At this time, the user selects an image to be transmitted to the SNS from a large number of images saved in the information processing apparatus. For example, when transmitting images related to one event, the user selects a collection of images related to this event and checks an image suitable for transmission from among them.

[0003] Patent Document 1 discloses that when determining a face image of a main person for each event from captured images obtained for each event, the face that appears most frequently in the captured image is regarded as the main person, and an electronic album with this main person as the protagonist is generated.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, as in Patent Document 1, the person with the face that appears most frequently in the captured image of an event is not necessarily the main person. For example, when the event is a birthday party in which a plurality of children participate, a child who is not the main character of the birthday party may appear more frequently in the captured image than a child who is the main character. In this case, there is also a possibility that a child who is not the main character is determined as the main person and becomes the protagonist.

[0006] Therefore, an object of the present invention is to enable efficient extraction of images containing key figures from an event when selecting images from a group of images related to that event. [Means for solving the problem]

[0007] To achieve the above objective, the information processing apparatus of the present invention is characterized by comprising: an acquisition means for acquiring information about an object for identifying a main person based on the type of event of a group of images including multiple images; a determination means for determining the main person from the images in the group of images that contain the object based on the information about the object acquired by the acquisition means; and an extraction means for extracting images from the group of images that contain the person determined by the determination means. [Effects of the Invention]

[0008] According to the present invention, when selecting images from a group of images related to an event, images containing the main people of that event can be efficiently extracted. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the configuration of the system according to this embodiment. [Figure 2] This is a diagram showing the configuration of the information processing device 101 on which the image management application according to this embodiment is running. [Figure 3] This is an example of a screen displayed in the image management application according to this embodiment. [Figure 4] This is an example screen showing how to view an image selected from a group of images in the image management application according to this embodiment. [Figure 5] This flowchart shows the process of selecting an image based on the main subject from a group of images in the image management application according to this embodiment. [Figure 6] This is a correspondence table between event types and characteristic objects according to this embodiment. [Modes for carrying out the invention]

[0010] This embodiment will be described below with reference to the drawings.

[0011] Figure 1 shows the system configuration according to this embodiment. User 110 uploads images taken with an imaging device 100 such as a digital camera to an image management application on an information processing device 101 via the Internet 102.

[0012] User 110 accesses the image management application on the information processing device 101 via the internet 102 using a PC 103 or mobile device 104 as a display device. The user uses the image management application to view images extracted from the image group uploaded and stored on the information processing device 101, and also sends images from the information processing device 101 to an external server (not shown), such as an SNS.

[0013] Figure 2 is a diagram showing the configuration of the information processing device 101 on which the image management application according to this embodiment operates. The information processing device 101 may be implemented as a single computer device, or its functions may be distributed among multiple computer devices as needed. If it is composed of multiple computer devices, they are connected by a communication line such as a Local Area Network (LAN) so that they can communicate with each other.

[0014] In Figure 2, the control unit 201 controls the entire information processing device 101, and is, for example, a Central Processing Unit (CPU).

[0015] Read-only memory (ROM) 202 stores programs, parameters, and libraries that do not require modification. Specifically, ROM 202 stores image analysis libraries, person detection libraries, object analysis libraries, etc. This image analysis library is used to analyze events such as when an image was taken. The object analysis library may be run on an external image analysis server outside of the information processing device 101. In that case, the information processing device 101 sends image files to the image analysis server via a communication line such as the Internet 102 and obtains image analysis results from the image analysis server.

[0016] Random Access Memory (RAM) 203 temporarily stores programs and data supplied from external devices. Specifically, RAM 203 stores the correspondence table shown in Figure 6, which contains information used by the image management application, and image files uploaded to the image management application by the user. ROM 202 and RAM 203 also include recording media (such as memory cards) for recording images.

[0017] The external storage device 204 is a hard disk or memory card fixed to the information processing device 101 and stores programs such as the Operating System (OS). The input interface (input I / F) 205 connects to input devices such as a pointing device or keyboard that receive user input and input data. The Bit Move Unit (BMU) 206 controls data transfer, for example, between memories (e.g., between VRAM 207 and other memories) and between memory and each I / O device (e.g., network interface 209).

[0018] The Video Random Access Memory (VRAM) 207 draws an image to be displayed on the display unit 211. The generated image is transmitted to the display unit 211 according to a predetermined rule, and thereby the display unit 211 displays information such as an image. The Network Interface (NET I / F) 209 is connected to a communication line 210 such as the Internet.

[0019] The transmission unit 212 has a function of transmitting the extracted image to the SNS.

[0020] The system bus 213 communicably connects each of the units 201 to 209 and 212. Also, the system bus 213 is connected to the Internet 102 via the NET I / F 209.

[0021] The external storage device 204 stores the program software of the image management application in the present embodiment or stores the program software of the image analysis application. Also, the external storage device 204 stores an image file, image management information, etc. transmitted from the imaging device 100 in FIG. 1 and the like.

[0022] In the image management application, for an image group composed of a plurality of images of a specific event, the type of event of the images included in the image group is analyzed, and based on the object characteristic of the event, the main person in the event is determined. Then, by extracting an image in which the main person and the characteristic object are captured together, the image to be transmitted to the SNS can be narrowed down and extracted.

[0023] Figure 3 shows an example of the image management screen 300 displayed in the image management application according to this embodiment. The image management screen 300 displays a list of thumbnail images of images uploaded by the user. The thumbnail images are 305-1 to 305-8. In addition, this image management screen 300 displays groups of images taken at the same event, including group A301 where the event is identified as "birthday" and group B302 where the event is identified as "sports (soccer)". For each image group, the shooting date 303 and event name 304 are displayed.

[0024] In each thumbnail image, a frame is superimposed over the characteristic object and main person identified by the image analysis results. This makes it easy to identify images that contain the characteristic object and main person. Thumbnail images 305-1 and 305-2 contain both the characteristic object and the main person, while thumbnail images 305-3 to 305-5 contain only the main person. Thumbnail image 305-6 contains neither the characteristic object nor the main person.

[0025] Furthermore, the image analysis library stored in ROM202 may be used to further analyze the images included in image group A, and the images to be displayed as image group A or B may be further narrowed down based on information such as whether the image is in focus, whether the eyes are closed, whether there is red-eye, and whether the images are similar. The number of images to display may be set in advance by the user.

[0026] When the user instructs the system to transition to the SNS sending confirmation screen, the sending confirmation screen 401 shown in Figure 4 is displayed. Specifically, when the user presses the confirmation button 306, the sending confirmation screen 401 is displayed. This sending confirmation screen 401 displays thumbnail images 305-1 and 305-2, which are images that include both a distinctive object and the main person. This allows the user to easily extract images to send to SNS. The number of images to be extracted may be set in advance by the user, or the number of images to be extracted may be changed depending on the destination (type of SNS) to which the extracted images are sent.

[0027] The submission confirmation screen 401 displays the face image of the person identified as the main person in image group A, along with guidance 411 indicating that the images were extracted with that person as the focus, and a person change button 412 for instructing the user to change the main person. Furthermore, the submission confirmation screen 401 also displays an image replacement button 413 for replacing the extracted images and a send button 414 for sending to social media.

[0028] Figure 5 is a flowchart showing the process of extracting images based on key individuals from a group of images of a specific event using an image management application. This process is executed by the control unit 201 of the information processing device 101, which reads correspondence tables and image files stored in RAM 203 according to instructions from the image management application program read from the external storage device 204.

[0029] In S501, the control unit 201 analyzes the metadata of multiple image files read from the RAM 203 (multiple image files uploaded to the information processing device 101 by the user). Specifically, the control unit 201 reads the date and time of shooting and the location of shooting from each image file and records them in the RAM 203.

[0030] In S502, the control unit 201 reads the date and time of capture and location information for each image file recorded in the RAM 203 in S501, and groups images related to the same event. Images to be included in the same event include images with consecutive capture dates and times, or images with capture dates and times within a predetermined range, or images taken in locations within a certain range. Images may also be grouped based on the type of imaging device 100 that captured the image, or information about the user 110 who took the image. In this embodiment, the group of images related to the same event is referred to as image group A.

[0031] In S503, the control unit 201 uses the image analysis library stored in ROM 202 to analyze each image in image group A and identify the type of event in image group A. The control unit 201 then associates image group A with the type of event in image group A and stores it in RAM 203. If the type of event in image group A is to be identified by analyzing whether each image in image group A contains a characteristic object that can identify the type of event, the correspondence table (Figure 6) stored in RAM 203 may be read and referred to. In that case, example 602 of the correspondence table in Figure 6, which identifies the type of event, is referred to. In this embodiment, the case in which the event in image group A is identified as "birthday" will be explained as an example.

[0032] In S504, the control unit 201 reads the correspondence table (Figure 6) stored in the RAM 203 and determines whether the events of image group A are included in the list of events 601 in the correspondence table in Figure 6. If it is determined that the events of image group A are included in the list of events 601 in the correspondence table in Figure 6, the process proceeds to S505; otherwise, the process of selecting an image based on the main person ends.

[0033] In the correspondence table in Figure 6, event 601 is a list of events in which a main character exists. If an event does not contain a main character, the image selection based on the main character, as in this embodiment, is not performed. In this embodiment, the control unit 201 determines that the event in image group A is a "birthday," and therefore the event in image group A is included in the list of events 601 in the correspondence table in Figure 6.

[0034] In S505, the control unit 201 obtains information about objects used to identify the main person associated with the event in image group A. Specifically, since the event in image group A is "birthday," the control unit 201 obtains "cake, candles, and banner," which is example 603 of objects (characteristic objects) used to identify the main person corresponding to "birthday."

[0035] In S506, the control unit 201 uses the object analysis library stored in the ROM 202 to analyze the objects depicted in the images included in image group A.

[0036] In S507, the control unit 201 identifies an image from image group A that contains a characteristic object corresponding to an event in image group A.

[0037] In S508, the control unit 201 uses the person detection library stored in ROM 202 to detect people who are pictured with characteristic objects in the images identified in S507, and adds those people to the list of candidate main people. The control unit 201 then records the list of candidate main people in RAM 203. Here, the people detected as candidate main people are those who are pictured within a predetermined range from the characteristic object in each image.

[0038] In S509, the control unit 201 determines one of the candidates for the main person detected in S508 as the main person of image group A and records that person. Multiple people may be detected as candidates for the main person. Therefore, based on predetermined conditions, one of the multiple people detected as candidates for the main person is determined to be the main person. Here, predetermined conditions include the person who appears the most often, the person who is in focus, and the person whose face is within a predetermined range from the center of the image.

[0039] In S510, the control unit 201 extracts images that include at least the main person. The extracted images are displayed on the transmission confirmation screen 401 in Figure 4. Alternatively, images that show the main person and a distinctive object together may be extracted, as shown on the transmission confirmation screen 401 in Figure 4.

[0040] <Other Embodiments> The various controls described above, which are performed by the CPU 106 of the information processing device 100, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0041] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0042] The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having the computer (or CPU, MPU, etc.) of that system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0043] Furthermore, the disclosure of this embodiment includes the following configurations and methods.

[0044] (Composition 1) An acquisition means for obtaining information about an object to identify a main person based on the type of event in a group of images containing multiple images, Based on the information about the object obtained by the acquisition means, a determination means for determining the main person from the image containing the object among the group of images, An extraction means for extracting images from the aforementioned group of images that include the person determined by the determination means, An information processing device characterized by having the following features.

[0045] (Configuration 2) The information processing apparatus according to configuration 1, characterized in that the extraction means extracts images from the group of images that include at least the main person.

[0046] (Composition 3) The information processing apparatus according to configuration 1 or 2, characterized in that the extraction means extracts images from the group of images that include both the main person and the object.

[0047] (Composition 4) The information processing apparatus according to any one of configurations 1 to 3, characterized in that it has a transmission means for transmitting the image extracted by the extraction means to an external server.

[0048] (Composition 5) The information processing apparatus according to any one of configurations 1 to 4, characterized in that the determination means determines that a person who is visible within a predetermined range from the object in the image group is the main person.

[0049] (Composition 6) The information processing apparatus according to any one of configurations 1 to 5, characterized in that, when a plurality of persons are detected as candidates for the principal person, the determination means determines one of the plurality of persons as the principal person based on predetermined conditions.

[0050] (Composition 7) The information processing apparatus according to configuration 6, characterized in that the predetermined conditions are that the person appears most frequently, the person is in focus, or the face of the main person is captured within a predetermined range from the center of the image.

[0051] (Composition 8) An acquisition step to obtain information about an object for identifying a key person based on the event type of a group of images containing multiple images, Based on the information about the object obtained in the acquisition step, a decision step is made to determine the main person from the image containing the object among the group of images, An extraction step is to extract images from the aforementioned group of images that contain the person determined in the determination step, A control method for an information processing device, characterized by having the following features.

[0052] (Composition 9) A program that causes a computer to execute the control method for the information processing device described in Configuration 8.

[0053] (Composition 10) A computer-readable recording medium containing a program that causes the computer to execute the control method for the information processing device described in Configuration 8.

Claims

1. An acquisition means for obtaining information about an object to identify a main person based on the type of event in a group of images containing multiple images, Based on the information about the object obtained by the acquisition means, a determination means for determining the main person from the image containing the object among the group of images, An extraction means for extracting images from the aforementioned group of images that include the person determined by the determination means, An information processing device characterized by having the following features.

2. The information processing apparatus according to claim 1, characterized in that the extraction means extracts images from the group of images that include at least the main person.

3. The information processing apparatus according to claim 2, characterized in that the extraction means extracts images from the group of images that include both the main person and the object.

4. The information processing apparatus according to claim 1, further comprising a transmission means for transmitting the image extracted by the extraction means to an external server.

5. The information processing apparatus according to claim 1, characterized in that the determination means determines that a person who is visible within a predetermined range from the object in the image group is the main person.

6. The information processing apparatus according to claim 1, characterized in that, when a plurality of persons are detected as candidates for the principal person, the determination means determines one of the plurality of persons as the principal person based on predetermined conditions.

7. The information processing apparatus according to claim 6, characterized in that the predetermined conditions are that the person appears most frequently, the person is in focus, or the face of the main person is captured within a predetermined range from the center of the image.

8. An acquisition step to obtain information about an object for identifying a key person based on the event type of a group of images containing multiple images, Based on the information about the object obtained in the acquisition step, a decision step is made to determine the main person from the image containing the object among the group of images, An extraction step is to extract images from the aforementioned group of images that contain the person determined in the determination step, A control method for an information processing device, characterized by having the following features.

9. A program for causing a computer to execute the control method for the information processing device described in claim 8.

10. A computer-readable recording medium containing a program for causing a computer to execute the control method of the information processing device described in claim 8.

Images