Information processing device, its control method, and program

Abstract

When a user selects multiple images from a group of images, they may not be able to select images that properly contain the subject. [Solution] The information processing device of the present invention includes: a receiving means for receiving instructions from a user to select an image from a plurality of images; a creation means for creating a group of images from the plurality of images based on the image selected by the instruction; a identification means for identifying a subject included in the images of the group of images; and a control means for controlling the device to notify information about the subject that is not depicted in the image selected by the instruction if the subject identified by the identification means is not depicted in the image.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method thereof, and a program.

Background Art

[0002] There is an information processing apparatus for managing images taken by electronic devices such as digital cameras and smartphones. For example, a user transmits and stores a taken image from an electronic device to an information processing apparatus on a network. Then, the information processing apparatus transmits the image to a social networking service (SNS). At this time, the user selects a large number of images stored in the information processing apparatus or an image to be transmitted to the SNS. For example, when transmitting images related to one event, the user selects a group of images related to this event and checks an image suitable for transmitting from them.

[0003] Patent Document 1 discloses an information processing apparatus that presents a user with a group of event photos related to posting content existing in a terminal by using the user's posting content (photos) to the SNS.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, even when using the information processing device described in Patent Document 1 to present a group of event photos to a user, the selected photo may not necessarily include all the people who participated in the event. In particular, when sending the selected photo to social media, users tend to prioritize photos where they look good, and may send a photo to social media without realizing that some of the event participants are not in the picture.

[0006] Therefore, the objective of the present invention is to enable users to select images that appropriately include the subject when selecting an image from a group of images. [Means for solving the problem]

[0007] To achieve the above objective, the information processing device of the present invention is characterized by comprising: a receiving means for receiving instructions from a user to select an image from a plurality of images; a creation means for creating a group of images from the plurality of images based on the image selected by the instruction; a identification means for identifying a subject included in the images of the group of images; and a control means for controlling the device to notify information about the subject that is not depicted in the image selected by the instruction if the subject identified by the identification means is not depicted in the image. [Effects of the Invention]

[0008] According to the present invention, when a user selects multiple images from a group of images, the present invention enables the selection of images that appropriately contain the subject. [Brief explanation of the drawing]

[0009] [Figure 1] This is an example of a block diagram illustrating the configuration of the information processing device 100 in this embodiment. [Figure 2] This is an example of an image registered in the information processing device 100 in this embodiment. [Figure 3] This is an example of the data configuration of the information processing device 100 in this embodiment. [Figure 4] This is a flowchart for registering images and image-related data in this embodiment. [Figure 5] This is a flowchart for notifying the user of a person who is not present in the image selected by the user in this embodiment. [Figure 6] This embodiment shows an example of a notification screen 600 that notifies the user of a person who is not present in the image selected by the user. [Modes for carrying out the invention]

[0010] This embodiment will be described below with reference to the drawings.

[0011] Figure 1 is an example of a block diagram illustrating an example configuration of the information processing device 100 in this embodiment. The information processing device 100 includes a display 101, VRAM 102, BMU 103, keyboard 104, PD 105, and CPU 106. Furthermore, the information processing device 100 includes a ROM 107, RAM 108, HDD 109, flexible disk 110, network I / F 111, transmission unit 112, and bus 113.

[0012] The display 101 shows, for example, icons, messages, menus, and other user interface information for managing the information processing device 100.

[0013] The VRAM (Video Random Access Memory) 102 is used to draw content for display on the display 101. The content data drawn to the VRAM 102 is transferred to the display 101 according to predetermined rules, and the content is then displayed on the display 101.

[0014] The BMU (Bit Move Unit) 103 controls, for example, data transfer between memories (e.g., between VRAM 102 and other memories) and data transfer between memory and each I / O device (e.g., network I / F 111).

[0015] The keyboard 104 has various keys for inputting characters and the like.

[0016] The PD (Pointing Device) 105 is used, for example, to indicate icons, menus, and other contents displayed on the display 101 or for dragging and dropping objects.

[0017] The CPU 106 controls each device based on control programs such as the OS stored in the ROM 107, HDD 109, or disk drive 110 and programs of the information processing apparatus 100 described later.

[0018] The ROM (Read Only Memory) 107 stores various control programs and data.

[0019] The RAM (Random Access Memory) 108 has a work area for the CPU 106, a data backup area during error processing, a load area for control programs, and the like. The ROM 107 and RAM 108 also include a recording medium (such as a memory card) for recording images.

[0020] The HDD (Hard Disk Drive) 109 stores data such as control programs executed in the information processing apparatus 100 and temporarily stored data. Further, when image data is transmitted from an external electronic device to the information processing apparatus 100, the HDD 109 stores and saves the transmitted image data.

[0021] The network I / F (NET I / F) 111 communicates with other information processing apparatuses, printers, etc. via a network.

[0022] The transmission unit 112 has a function of transmitting an image selected by the user to the SNS.

[0023] The bus 113 includes an address bus, a data bus, and a control bus.

[0024] The control program for the CPU 106 can be provided from the ROM 107, HDD 109, and flexible disk 110, or from other information processing devices via the network interface 111.

[0025] The operating component may be a touch panel instead of the keyboard 104 or PD (pointing device) 105.

[0026] Figure 2 shows an example of an image stored in the information processing device 100 in this embodiment. When image data is transmitted from an external electronic device such as a smartphone via the network I / F 111, the CPU 106 stores this image data in the HDD 109. Figure 2 shows an example of an image stored in the HDD 109 in this way. In Figure 2, there are five images stored in the information processing device 100: images 201 to 205. Image 201 is an image of person 211 (file name "000001.JPG"). Image 202 is an image of person 212 (file name "000002.JPG"). Image 203 is an image of person 213 (file name "000003.JPG"). Image 204 is an image of person 211 and person 212 (file name "000004.JPG"). Image 205 is an image (filename "000005.JPG") of person 214.

[0027] Figure 3 shows an example of the data configuration of the information processing device 100 in this embodiment. The CPU 106 manages image data and other data stored in the HDD 109. Figure 3(a) shows an example of image-related data stored in the information processing device 100 by the CPU 106, and this data is generated by the CPU 106. 300 shows a management table for the data related to the stored images. Hereinafter referred to as DB300. 301 shows the unique image ID of the stored image. The image IDs corresponding to images 201 to 205 in Figure 2 are image IDs 201 to 205, respectively. 302 shows the file name of the stored image. 303 shows the date and time the stored image was taken. 304 shows the ID of the person in the stored image. The person IDs corresponding to people 211 to 214 in Figure 2 are person IDs 211 to 214, respectively.

[0028] Record 305 corresponds to the record for image 201 in Figure 2, record 306 corresponds to the record for image 202 in Figure 2, and record 307 corresponds to the record for image 203 in Figure 2. Additionally, record 308 corresponds to the record for image 204 in Figure 2, and record 309 corresponds to the record for image 205 in Figure 2.

[0029] Figure 3(b) shows an example of data for people in an image stored in the information processing device 100. 310 shows the management table for people in the stored image. Hereafter, this will be referred to as DB310. 311 shows the unique person ID of each person in the stored image. The person IDs corresponding to people 211-214 in Figure 2 are person IDs 211-214, respectively. 312 shows the feature quantities of the people in the stored image. 313 shows clip data of the face portion of each person in the stored image. 314 shows the vertical and horizontal dimensions of the clip data of the face portion of each person in the stored image.

[0030] 315 corresponds to the record relating to person 211 in Figure 2, and 316 corresponds to the record relating to person 212 in Figure 2. 317 corresponds to the record relating to person 213 in Figure 2, and 318 corresponds to the record relating to person 214 in Figure 2.

[0031] Figure 4 is a flowchart illustrating the process of generating or saving images and image-related data in this embodiment. In this embodiment, the process of saving images is described as being performed in the order of image 201, image 202, image 203, image 204, and image 205 shown in Figure 2. Furthermore, when the information processing device 100 receives an image transmitted by a user from an external electronic device, the flowchart in Figure 4 is initiated.

[0032] In S401, the CPU 106 stores the received image data in the HDD 109 and retrieves the file name, shooting date and time, and person information from this image data. There are various publicly known techniques for retrieving person information, and the type of technique used to retrieve person information in this process is not limited.

[0033] In S402, CPU 106 obtains feature quantities, clip data, and the width and height of the clip data from the person information acquired in S401. In this embodiment, the feature quantities are feature information representing facial features extracted from the face image by a predetermined extraction process. The process of extracting feature quantities from the face image is a process using a neural network trained using deep learning. CPU 106 also obtains clip data by cropping the face portion of the person in the registered image, and obtains the width and height of the acquired clip data.

[0034] In S403, CPU106 determines whether the person's features that match those obtained in S402 are registered in DB310. If it determines that the person's features that match those obtained in S402 are registered in DB310, it proceeds to S405; otherwise, it proceeds to S404.

[0035] In S404, CPU106 generates a person ID for an unregistered person and registers the feature quantities, clip data, and the width and height of the clip data, which correspond to the person ID, in DB310. At the time when the registration process for each of images 201 to 203 is executed, data for each of persons 211 to 213 has not yet been registered, so person IDs 211 to 213 are generated and the information corresponding to those person IDs is registered.

[0036] In S405, CPU 106 determines which size is larger: the width and height of the clip data acquired in S402 or the width and height of the clip data registered in DB310. If it is determined that the width and height of the clip data acquired in S402 is larger, the process proceeds to S406. If it is determined that the width and height of the clip data registered in DB310 is larger, the process proceeds to S407. At the time the registration process for image 204 is executed, data for people 211 and 212 has already been registered. Therefore, the process determines which size is larger: the width and height of the clip data acquired in S402 or the width and height of the clip data registered in DB310.

[0037] In S406, CPU 106 updates the clip data and its width and height to match the clip data and its width and height to match those obtained in S402. For example, suppose it is determined that the width and height of the clip data for person 211 in image 204 is larger than the width and height of the clip data for person 211 registered in DB310. In this case, the clip data and its width and height for person 211 will match the width and height of the clip data for person 211 in image 204. As a result, as shown in Figure 6, when displaying clip data registered in DB310 for people not present in the image selected by the user, clip data that makes it easier to identify the person will be displayed.

[0038] In S407, CPU 106 determines whether there are any unprocessed individuals based on the person information obtained in S401. That is, if person information is obtained indicating that there are two or more people in the image, it determines whether there are any unprocessed individuals. If there are no unprocessed individuals, it proceeds to S408; otherwise, it proceeds to S402.

[0039] In S408, CPU106 generates an image ID for the captured image and registers the person ID that matches the generated image ID with the file name, shooting date and time, and person information obtained in S401 in DB300.

[0040] Figure 5 is a flowchart of the process for notifying the user of a person not present in the image selected by the user in this embodiment. This flowchart is a process that starts when the user selects an image (for example, when the user presses a selection button).

[0041] In S501, the CPU 106 accepts the user's selection of an image from among the images stored in the information processing device 100.

[0042] In S502, the CPU 106 determines an image group from the image data stored in the HDD 109, including the image selected in S501 or S508, and multiple images that are close in date and time to the selected image. In other words, based on the image selected in S501, the CPU 106 selects images from the image data stored in the HDD 109 that meet specific conditions, such as the date and time of shooting, to create an image group. This makes it possible to create an image group of images taken at events such as trips or reunions that are close in date and time. Images that are hidden by filters, etc., may also be added to the image group if they meet the specific conditions.

[0043] For example, images taken within a predetermined range (e.g., within 5 minutes before or after) of the date and time of the image selected in S501 are grouped together. In this case, if the user selects image 202, an image group is created containing both image 201 and image 202, since image 201 is taken within 5 minutes before or after the date and time of image 202. Subsequently, if the user selects image 204, image 203, taken between the date and time of image 201 and the date and time of image 204, is also added to the image group. In this way, if the user selects two or more images, the image group will include images taken between the earliest and latest date and time of the selected images. Note that by setting a specific condition to within 0 seconds before or after the date and time of the shooting, images close to the selected image may not be included in the image group. Furthermore, the specific condition is not limited to the date and time of shooting, but may also be a condition related to the file name.

[0044] In S503, CPU 106 determines whether or not there are any unprocessed images in the image group created in S502. For example, if S501 accepts the user's selection of images 201 and 204, and S502 creates an image group of images 201 to 204, if it is determined that there are any unprocessed images in that image group, the process proceeds to S504; otherwise, it proceeds to S508.

[0045] In S504, CPU106 retrieves a record related to an image that matches an unprocessed image. Specifically, when processing images 201 through 204 in that order, if all images 201 through 204 are unprocessed, the processing will be performed in that order. If the unprocessed image is image 201, record 305 related to image 201, which corresponds to image ID 201 of image 201, is retrieved.

[0046] In S505, CPU 106 adds the person IDs from the records obtained in S504 to set A. Specifically, when it obtains record 305 related to image 201, it adds person ID 211 to set A. At the time image 201 is processed, set A is empty, so it adds person ID 211 to set A. Here, set A is a collection of person IDs corresponding to people who participated in the event, and it does not have duplicate values; adding values ​​that are already held is ignored. Therefore, at the time image 204 is processed, person ID 211 already exists in set A, so adding person ID 211 is ignored. Thus, when processing of image 204 is finished, set A will contain person IDs 211, 212, and 213, and people 211, 212, and 213 are all members who participated in the event. In this way, in S505, CPU 106 identifies the people who are subjects included in the set of images.

[0047] In S506, CPU106 determines whether the unprocessed image is the image selected by the user in S501. If it is determined to be the image selected by the user in S501, the process proceeds to S507; otherwise, it proceeds to S503.

[0048] In S507, CPU3 adds the person ID of the record obtained in S504 to set B. Then, it proceeds to S508 with the unprocessed image treated as a processed image. When record 305 related to image 201 is obtained, person ID 211 is added to set B. At the time image 201 is processed, set B is empty, so person ID 211 is added to set B. Here, set B is a collection of person IDs corresponding to the people in the image selected by the user, and it does not have duplicate values, and adding values ​​that are already held is ignored. For example, if the user selects images 201 and 204 in S501, at the time image 204 is processed, person ID 211 already exists in set B, so adding person ID 211 is ignored. Then, when the processing of image 204 is finished, set B will contain person ID 211 and person ID 212, and person 211 and person 212 are members appearing in the image selected by the user.

[0049] In S508, CPU3 determines whether to accept the user's input to select another image. If the user's input to select another image is accepted, the process proceeds to S502; otherwise, it proceeds to S509. If the user's input to select another image is accepted, the image selected in S508 and the image selected based on the image selected in S508 are added to the group of images created based on the image selected in S501.

[0050] In S509, CPU106 calculated the set difference between set A and set B. The values ​​of set A are person ID 211, person ID 212, and person ID 213, and the values ​​of set B are person ID 211 and person ID 212. Therefore, the value of the set difference is person ID 213. In other words, a value remains in the set difference. This indicates that the image selected by the user does not contain a person who was identified as a subject participating in the event from the image set, and that person is person 213, corresponding to person ID 213.

[0051] In S510, CPU106 determines whether or not there are any values ​​remaining in the difference set. If it is determined that there are values ​​remaining in the difference set, the process proceeds to S511; otherwise, the process of notifying the user of people who are not present in the image selected by the user ends.

[0052] In S511, CPU106 retrieves records from DB310 related to person IDs whose values ​​remain in the set difference. Specifically, the set difference value calculated in S509 is person ID 213, and record 317 related to person ID 213 is retrieved.

[0053] In S512, the CPU 106 obtains clip data from the record acquired in S511 and displays it on the display 101 of the information processing device 100. Specifically, it obtains clip data for person 213 from record 317 related to person ID 213 and displays the clip data for person 213 on the display 101. Figure 6 is an example of a notification screen 600 that notifies the user of a person not present in the image selected by the user in this embodiment. This notification screen 600 embeds the clip data for person 213 in area 601 and notifies the user of a person not present in the image selected by the user. In particular, by displaying the person's face on the notification screen 600, it is possible to determine who the person is who is not present in the selected image, even if information identifying the person, such as their name, is not recorded.

[0054] In S513, if the CPU 106 receives a user input to the re-select button 603 on the notification screen 600 displayed in S512, it proceeds to S501 while retaining the values ​​of the image group, set A, and set B. This re-select button 603 is a button that transitions to a screen where images can be re-selected. By displaying people not included in the image selected by the user on the notification screen 600, the user can easily select the desired image even when re-selecting an image.

[0055] Furthermore, in S513, if the CPU 106 receives an operation from the user on the notification screen 600 displayed in S512 by pressing the confirm button 602, it terminates the process of notifying the user of a person who is not shown in the image selected by the user.

[0056] <Other Embodiments> The various controls described above, which are performed by the CPU 106 of the information processing device 100, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0057] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0058] The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having the computer (or CPU, MPU, etc.) of that system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0059] Furthermore, the disclosure of this embodiment includes the following configurations and methods.

[0060] (Composition 1) A means for receiving instructions from the user to select an image from multiple images, A creation means for creating an image group from the plurality of images based on the image selected according to the above instructions, A means for identifying a subject included in the images of the aforementioned group of images, If the subject identified by the identifying means is not visible in the image selected by the above instruction, the control means controls the notification of information regarding the missing subject. An information processing device characterized by having the following features.

[0061] (Configuration 2) The information processing apparatus according to Configuration 1, characterized in that the control means controls not to notify the information if the subject identified by the identification means is captured in the image selected by the instruction.

[0062] (Composition 3) The information processing apparatus according to configuration 1 or 2, characterized in that the creation means creates a group of images including images whose shooting dates and times are within a predetermined range, based on the shooting date and time of the image selected by the instruction.

[0063] (Composition 4) The information processing apparatus according to any one of configurations 1 to 3, characterized in that the subject is a person.

[0064] (Composition 5) The information processing device according to configuration 4, characterized in that the control means controls to notify information about the face of a person if the person identified by the identification means is not depicted in the image selected by the instruction.

[0065] (Composition 6) The information processing apparatus according to configuration 4 or 5, characterized in that the identifying means identifies a person based on the facial feature quantities of a person depicted in an image included in the group of images.

[0066] (Composition 7) The information processing apparatus according to any one of configurations 1 to 6, characterized in that, if the subject identified by the identification means is not depicted in the image selected by the instruction, the control means displays a button for transitioning to a screen on which the user can re-select the image, along with information about the subject that is not depicted.

[0067] (Composition 8) A reception step that accepts the user's selection of an image, A creation step in which a group of images is created based on the images selected in the above acceptance step, A selection step to identify the subject included in the aforementioned group of images, If the image selected in the reception step does not contain the subject identified in the identification step, a control step is performed to control the system to notify information about the subject. A control method for an information processing device, characterized by having the following features.

[0068] (Composition 9) A program that causes a computer to execute the control method for the information processing device described in Configuration 8.

[0069] (Composition 10) A computer-readable recording medium containing a program that causes the computer to execute the control method for the information processing device described in Configuration 8.

Claims

1. A means for receiving instructions from the user to select an image from multiple images, A creation means for creating an image group from the plurality of images based on the image selected according to the above instructions, A means for identifying a subject included in the images of the aforementioned group of images, If the subject identified by the identifying means is not visible in the image selected by the above instruction, the control means controls the notification of information regarding the missing subject. An information processing device characterized by having the following features.

2. The information processing apparatus according to claim 1, characterized in that the control means controls not to notify the information if the subject identified by the identification means is captured in the image selected by the instruction.

3. The information processing apparatus according to claim 1, characterized in that the creation means creates a group of images including images whose shooting dates and times are within a predetermined range, based on the shooting date and time of the image selected by the instruction.

4. The information processing apparatus according to claim 1, characterized in that the subject is a person.

5. The information processing apparatus according to claim 4, characterized in that the control means controls to notify information regarding the face of a person if the person identified by the identification means is not depicted in the image selected by the instruction.

6. The information processing apparatus according to claim 4, wherein the identifying means identifies a person based on the facial feature quantities of a person depicted in an image included in the group of images.

7. The information processing apparatus according to claim 1, wherein if the subject identified by the identification means is not depicted in the image selected by the instruction, the control means displays information about the subject that is not depicted, along with a button to transition to a screen on which the user can re-select the image.

8. A reception step that accepts the user's selection of an image, A creation step in which a group of images is created based on the images selected in the above acceptance step, A selection step to identify the subject included in the aforementioned group of images, If the image selected in the reception step does not contain the subject identified in the identification step, a control step is performed to control the system to notify information about the subject. A control method for an information processing device, characterized by having the following features.

9. A program for causing a computer to execute the control method for the information processing device described in claim 8.

10. A computer-readable recording medium containing a program for causing a computer to execute the control method of the information processing device described in claim 8.

Images