Image processing device, image processing method, and recording medium storing image processing program
Through the recognition and combination technology of the image processing device, the problem of identifying and combining images of different characters and different age groups in image processing is solved, and accurate image organization and retrieval effects are achieved.
Patent Information
- Application Number
- CN202010532731.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-06-28
- Filing Date
- 2020-06-11
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-06-11
AI Technical Summary
Existing image processing technologies are difficult to accurately identify and combine images of different characters, especially images of parents and children at different ages, and cannot effectively search and organize images.
An image processing device and method are designed to identify and estimate the age of a person by reading multiple images, determine whether it is the same person, and combine images when the age difference is within the threshold range, including character recognition, age estimation, The determination and combination detection mechanism can identify and combine images of different age groups.
Accurate identification and combination between images of different characters and different age groups, and finding images like parents' youth and children's youth is improved, improving the efficiency and effect of image organization and retrieval.
Smart Images

Figure CN112148910B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an image processing device, an image processing method, an image processing program and a recording medium storing the program. Background Art
[0002] When making a photo album with a specified person as the protagonist, even if it is the same person, there will be changes in the face, so sometimes it will not be recognized as the same person. Therefore, it is possible to consider a technology that can accurately identify a person by following the changes in the face over time (Patent Document 1). In addition, it is also possible to consider a technology that can perform efficient image retrieval when searching for images of people by classifying individual identification information in person units (Patent Document 2). In addition, it is also possible to consider a technology that selects images that match the theme of the album from multiple images collected from multiple users (Patent Document 3).
[0003] Patent Document 1: Japanese Patent Application Publication No. 2015-069495
[0004] Patent Document 2: Japanese Patent Application Publication No. 2010-079519
[0005] Patent Document 3: Japanese Patent Application Publication No. 2018-005843
[0006] When two or more images are to be sorted out from a large number of images, they are often sorted by images with a close shooting date and time, images shot during a specific event, favorite images, etc. However, if, for example, a father's childhood image and his child's childhood image can be compared, the user can discover new insights and new surprises. Therefore, sometimes the user re-watches the large number of images shot or has new exchanges based on the images shot, which sometimes promotes the use of the large number of images.
[0007] The invention described in the previous document 1 is an invention for accurate person recognition, and therefore does not consider finding images of parents and children in the same age group. The invention described in the previous document 2 is an invention for efficient image retrieval, and therefore does not consider finding images of parents and children in the same age group. The invention described in the previous document 3 is an invention for selecting images that match the theme of the album, and still does not consider finding images of parents and children in the same age group. Summary of the invention
[0008] The purpose of the present invention is to find images of different persons taken at the same age, such as images of parents in their childhood and images of the parents' children in their childhood.
[0009] The image processing device according to the invention comprises: a reading mechanism for reading an image from a recording device recording a plurality of images accessible to a user; a first image determining mechanism for determining any one of the plurality of images as the first image; a first person recognizing mechanism for recognizing a first person photographed in the first image; a first age estimating mechanism for estimating the age of the first person based on the shooting date and time of the first image; a second image determining mechanism for determining any one of the plurality of images except the first image as the second image; a second person recognizing mechanism for recognizing a second person photographed in the second image; a second age estimating mechanism for estimating the age of the first person based on the shooting date and time of the first image; a mechanism for estimating the age of a second person based on the shooting date and time of the second image; a person determining mechanism for determining whether the identified first person and the identified second person are the same person; an age difference determining mechanism for determining whether the difference between the estimated age of the first person and the estimated age of the second person is within a threshold value; and an image combination detecting mechanism for detecting the first image and the second image as an image combination when the person determining mechanism determines that the first person and the second person are not the same person and the age difference determining mechanism determines that the difference between the age of the first person and the age of the second person is within a threshold value.
[0010] The invention also provides an image processing method. That is, in this method, a reading unit reads an image from a recording device that records a plurality of images that a user can access, a first image determination unit determines any one of the plurality of images as a first image, a first person recognition unit recognizes a first person photographed in the first image, a first age estimation unit estimates the age of the first person at the shooting date and time of the first image, a second image determination unit determines any one of the plurality of images other than the first image as a second image, a second person recognition unit recognizes a second person photographed in the second image, and a second age estimation unit estimates the age of the first person. The age of the second person at the shooting date and time of the second image, the person judging mechanism judges whether the identified first person and the identified second person are the same person, the age difference judging mechanism judges whether the difference between the estimated age of the first person and the estimated age of the second person is within a threshold value; when the person judging mechanism judges that the first person and the second person are not the same person and the age difference judging mechanism judges that the difference between the age of the first person and the age of the second person is within a threshold value, the image combination detection mechanism detects the first image and the second image as an image combination.
[0011] The present invention also provides a computer program for controlling an image processing device and a recording medium storing the program.
[0012] Furthermore, the image processing device may be as follows: it comprises: a reading mechanism that reads an image from a recording device that records a plurality of images that can be accessed by a user; and a processor that determines any one of the plurality of images as a first image, identifies a first person captured in the first image, estimates the age of the first person based on the shooting date and time of the first image, determines any one of the plurality of images other than the first image as a second image, identifies a second person captured in the second image, estimates the age of the second person based on the shooting date and time of the second image, determines whether the identified first person and the identified second person are the same person, determines whether the difference between the estimated age of the first person and the estimated age of the second person is within a threshold value, and when it is determined that the first person and the second person are not the same person and the difference between the age of the first person and the age of the second person is within a threshold value, detects the first image and the second image as an image combination.
[0013] The second image specifying means specifies, as the second image, an image whose shooting date is separated from the shooting date of the first image by a threshold or more, for example.
[0014] Furthermore, the second image determination mechanism can determine, as the second image, an image that was captured in a year different from the year in which the first image was captured and in the same period as the shooting period of the first image (for example, the shooting season, the shooting month), or can determine, as the second image, an image that has the same composition as the first image.
[0015] In addition, the second image determination unit may determine as the second image an image shot at the same location as the shooting location of the first image, or may determine as the second image an image including at least one of the relatives or in-laws of the person included in the first image.
[0016] In addition, the second image specifying means specifies, as the second image, an image captured in the same event as the event in which the first image was captured, for example.
[0017] The image processing device may further include a control unit that controls the image processing device so as to repeat at least one of the identification of the first image and the identification of the second image until at least one image combination is detected.
[0018] The apparatus may further include: a similarity calculation unit that calculates at least one of a first similarity between a first person and a second person or a second similarity between a first image and a second image. In this case, for example, when the first similarity or the second similarity is greater than a threshold value, the image combination detection unit detects the first image and the second image as an image combination.
[0019] For example, a plurality of images of relatives or in-laws of a specific person are recorded in the recording device.
[0020] The device may further include a display control unit that controls the display device so as to display the first image and the second image adjacent to each other.
[0021] The device may further include a printer control unit that controls the printer so as to print the first image and the second image adjacent to each other.
[0022] The first image specifying means specifies the first image based on, for example, an instruction from a user.
[0023] Effects of the Invention
[0024] Although the first person included in the detected first image and the second person included in the second image are not the same person, the age difference when shooting is within the threshold range. Therefore, for example, it is possible to find images of different people taken in the same age group, such as images of parents' teenage years and images of the parents' children's teenage years. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is an overview of the image processing system.
[0026] Figure 2 It is a block diagram showing the electrical configuration of a storage server.
[0027] Figure 3 This is an example of an image stored on a hard disk.
[0028] Figure 4 It is a flowchart showing the processing flow of the storage server.
[0029] Figure 5 It is a flowchart showing the processing flow of the storage server.
[0030] Figure 6 It is a flowchart showing the processing flow of the storage server.
[0031] Figure 7 This is an example of an image stored on a hard disk.
[0032] Figure 8 This is an example of an image stored on a hard disk.
[0033] Fig. 9 This is an example of printed matter.
[0034] Fig.10 This is a flowchart showing the flow of the second image determination process.
[0035] Fig.11 This is a flowchart showing the flow of the second image determination process.
[0036] Fig.12 This is a flowchart showing the flow of the second image determination process.
[0037] Fig.13 This is a flowchart showing the flow of the second image determination process.
[0038] Fig.14 This is a flowchart showing the flow of the second image determination process.
[0039] Fig.15 This is a flowchart showing the flow of the second image determination process.
[0040] Fig.16 This is a flowchart showing the flow of the second image determination process.
[0041] Fig.17 This is a flowchart showing the flow of the second image determination process.
[0042] Fig.18 This is a flowchart showing the flow of the second image determination process.
[0043] Fig.19 This is a flowchart showing the flow of the image combination determination process. DETAILED DESCRIPTION
[0044] [First embodiment]
[0045] Figure 1 1 is a diagram showing an embodiment of the present invention, and shows an overview of an image processing system. In the following, the age group (age layer) proximity is explained in terms of the age difference being within a threshold range.
[0046] The image processing system includes n (n is plural) client computers 1-n, which are connected to the Internet. In addition, the image processing system also includes a storage server 10 and a printer server 20, each of which can communicate with each of the n client computers 1-n via the Internet.
[0047] The client computer 1-n sends an image file representing an image captured by the user to the storage server 10, and the storage server 10 receives and stores the image file sent from the user. The printer server 20 prints the first image and the second image determined as described below adjacently (refer to Fig. 9 ).
[0048] exist Figure 1 Although n client computers 1-n are shown in the figure, only one client computer may be included in the image processing system. The number of client computers is irrelevant as long as multiple users can use the client computers to upload image data to the storage server 10. Furthermore, communication devices such as smart phones may be used instead of the client computers 1-n.
[0049] Furthermore, the image processing system may be configured without using the printer server 20 .
[0050] Figure 2 It is a block diagram showing the electrical configuration of the storage server 10 .
[0051] The overall operation of the storage server 10 (an example of an image processing device) is centrally controlled by a CPU (Central Processing Unit) 11 .
[0052] The storage server 10 includes a communication device 12 for connecting to the Internet, a memory card reader / writer 13 for accessing a memory card 14, a micro-disc drive 15 for reading data recorded in a micro-disc 16, a hard disk 18 for recording image files sent from a user, and a hard disk drive 17 for writing image files in the hard disk 18 and reading image files from the hard disk 18. In addition, the storage server 10 also includes a memory 19 and a display device 19A.
[0053] The mini optical disk 16 (portable recording medium) stores a program for controlling the operation of the storage server 10, and the program recorded in the mini optical disk 16 is read by the mini optical disk drive 15 and installed in the storage server 10. Instead of reading the program from the mini optical disk 16 and installing it in the storage server 10, the program may be received in the storage server 10 via a network such as the Internet, and the received program may be installed in the storage server 10. Similar to the storage server 10, the printer server 20 also includes a CPU, a memory, a hard disk, a communication device, and the like.
[0054] Figure 3 A plurality of images stored in the hard disk 18 are shown.
[0055] The hard disk 18 stores a plurality of images of a plurality of users, such as images P1 to P100 of the user U1, images P101 to P200 of the user U2, and images P201 to P300 of the user U3. Images of users other than the users U1, U2, and U3 are also stored in the hard disk 18. The user U1 can access the images P1 to P100 of the user U1 by using the user ID and password, but cannot access the images P101 to P200 of the user U2 or the images P201 to P300 of the user U3. Other users are managed so that they can access their own images but cannot access the images of other users.
[0056] In this embodiment, an image of a user and images of the user's parents, children, older brothers, younger sisters, etc., taken at a similar age to the user's age at the time of the photo are found. For example, an image of a user at birth and an image of the user's father at birth are found, or an image of a user's elementary school entrance ceremony and an image of the user's father at the elementary school entrance ceremony are found. In this embodiment, user U1 is set as a user, but other users U2, U3, etc. may also be set as a user.
[0057] Figures 4 to 6 It is a flowchart showing the flow of processing of the storage server 10 (image processing device).
[0058] For example, the user U1 sends an image reading command from the client computer 1 to the storage server 10, and the image reading command is received in the storage server 10. In this way, the image files representing the images P1 to P100 of the user U1 stored in the hard disk 18 are read by the hard disk drive 17 (an example of a reading mechanism). Figure 4 The CPU 11 (an example of the first image determining means) determines the first image ( Figure 4 Step 32).
[0059] The first image may be the first image captured among the images P1 to P100, or any image. It is also possible to find images containing people among the images P1 to P100 and determine the first image from among the images containing people. It is also possible to send an image file representing the images P1 to P100 or an image file representing an image containing people among the images P1 to P100 from the storage server 10 to the client computer 1 used by the user U1, and display the images P1 to P100 or the images containing people among the images P1 to P100 on the display screen of the display device included in the client computer 1, and let the user U1 select the first image using the mouse of the client computer 1 (an example of the first image determination unit), thereby determining the first image. When the user U1 selects the first image, the identification data of the first image is sent from the client computer 1 to the storage server 10 through the communication device of the client computer 1.
[0060] When the first image is identified by the CPU 11 of the storage server 10 or the user U1, the CPU 11 (an example of the first person recognition unit) recognizes (detects) a person (the first person) included in the first image ( Figure 4 If the first image includes a plurality of persons, the plurality of persons are recognized by the CPU 11. Then, the CPU 11 (an example of the first age estimation means) estimates the age of the first person when the first image was taken ( Figure 4Step 34). When data indicating age is recorded in the header of the image file representing the first image, the age can be known from the data, and images of multiple people of known ages can be read in and the age of the first person can be estimated using AI (artificial intelligence) having a learning model that has learned the relationship between face, body shape, clothing, etc. and age (for example, as described in Japanese Patent Publication No. 2010-113526, Japanese Patent Publication No. 2010-257409, etc.). When there is information about the age of the first person in the first image (for example, the first person has a name tag with the school year recorded on it, etc.), the age of the first person is estimated by CPU 11 using the information.
[0061] Figure 7 and Figure 8 A portion of images P1 to P100 of the user U1 is shown. Figure 7 and Figure 8 , some images are shown by shooting year.
[0062] refer to Figure 7 Image P1 was taken in 1962, and is an image of the birth of user U1's father U11. Image P12 was taken in 1969, and is an image of user U1's father U11 entering elementary school. Image P12 shows user U1's father U11 and user U1's grandmother U21. Image P33 was taken in 1975, and is an image of user U1's father graduating from elementary school. Image P33 also shows user U1's father U11 and user U1's grandmother U21. Image P44 was taken in 1983, and is an image of user U1's father's coming-of-age ceremony. Image P51 was taken in 1990, and is an image of user U1 when he was born, and shows user U1.
[0063] refer to Figure 8 , image P51 and Figure 7 The image P51 shown is the same as that of the user U1. The image P72 was taken in 1997 and is an image of the user U1 entering elementary school. The user U1 and the user U1's mother U10 are photographed in the image P72. The image P83 was taken in 2003 and is an image of the user U1 graduating from elementary school. The user U1 and the user U1's mother U10 are also photographed in the image P83. The image P94 was taken in 2011 and is an image of the user U1's coming-of-age ceremony. The image P100 was taken in 2019 and is an image of the user U1's child U31 when he was born. The user U1's child U31 is photographed.
[0064] When data about the shooting year is recorded in the header of the image file representing the images P1 to P100, the shooting year of the images P1 to P100 can be known from the data. When the data about the shooting year is not recorded in the header of the image file, the shooting year can be estimated by analyzing the image. For example, when the shooting date is written in the image using the shooting date and time writing function during shooting, the shooting date can be obtained by analysis. When the shooting date is not written, the shooting year can be estimated based on the comparison of landmark buildings, the degree of fading of the photo and the digitization date and time, etc. In addition, multiple images with known shooting dates are read in, and the relationship between the existence of landmark buildings, the difference between the degree of fading of the photo and the digitization date and time, the clothing, hairstyle, and items carried by other people reflected in the image, etc., and the shooting date can be learned, thereby making a learning model. The shooting year can also be estimated using AI with the learning model (for example, as recorded in Japanese Patent Publication No. 2010-026713, Japanese Patent Publication No. 2011-525017, etc.).
[0065] Thus, the shooting date and time mentioned in the present invention also includes the estimated date and time. The estimated date and time includes the estimated date and time when the analog film print is digitized. The recording location of the estimated date and time can be a specified area of Exif (Exchangeable image file format) or an area unique to various image management software or applications. In short, as long as it is the confirmed or estimated shooting date and time of each image and the date and time that can be accessed by the device of the present invention, it will suffice.
[0066] For example, suppose that the CPU 11 or the user U1 determines Figure 8 The image P72 of the elementary school entrance ceremony of the user U1 is shown as the first image. The user U1 and the mother U10 of the user U1 are photographed in the image P72, so an image is found in which a person is photographed at the same age as the user U1 or the mother U10 of the user U1 in the image P72 of the elementary school entrance ceremony and a person other than the user U1 or the mother U10 of the user U1 is photographed.
[0067] refer to Figure 5 , the CPU 11 (an example of the second image identifying means) identifies the second image other than the first image P72 from the images P1 to P100 of the user U1 (step 35). As with the first image, the CPU 11 (an example of the second image identifying means) identifies the person (the second person) included in the second image ( Figure 5 The CPU 11 (an example of the second age estimation means) estimates the age of the second person ( Figure 5Step 37). If the header of the image file representing the second image contains data on age, the age of the second person can be determined based on the data. If the header of the image file does not contain data on age, the age can be estimated by analyzing the second image by CPU 11. The age estimation mentioned in the present invention can be performed using any of the following three methods. (1) If the person projected has been identified and the shooting date and time of the image and the birthday of the person are known, the age can be accurately calculated (this is also included in the meaning of the term "age estimation" in the present invention). (2) If the birthday is known and the shooting year can be estimated by the above-mentioned AI, the age can be estimated. (3) The age can be estimated by the above-mentioned AI based on the face, height (estimated by comparison with other people, etc.) and clothing of the projected person.
[0068] Then, the CPU 11 (an example of a person determination unit) determines whether the first person and the second person are the same person ( Figure 5 Step 38). If it is not the same person (in Figure 5 If "No" is returned in step 38 of the procedure, the CPU 11 (an example of an age difference determination unit) determines whether the difference between the age of the first person (the first age) and the age of the second person (the second age) is within a threshold value ( Figure 5 Step 39). When both the first person and the second person are multiple and the number is the same, when the multiple first persons and the multiple second persons are all the same, they are judged to be the same person. When both the first person and the second person are multiple but the number of the first person and the number of the second person are different, the first person and the second person will not be judged to be the same person even if the same person is included in a part. When either the first person or the second person is multiple and either the other is one person, the first person and the second person will not be judged to be the same person.
[0069] Furthermore, if the difference between the first age and the second age is within a threshold value (e.g., 2 years), the CPU 11 (an example of an image combination detection unit) determines (detects) the combination of the first image and the second image ( Figure 5 In step 40 of the embodiment of the present invention, when at least one of the first person and the second person is a plurality of persons, the CPU 11 determines whether the difference between the first age and the second age is within a threshold value in all combinations of the plurality of persons as the first person and the plurality of persons as the second person. As long as the difference between the first age and the second age is within the threshold value in one combination, the CPU 11 determines that the combination is the first image and the second image.
[0070] refer to Figure 8, if the first image is determined by the CPU 11 or the user U1 as the image P72 taken when the user U1 entered the elementary school and the image P83 is determined by the CPU 11 as the second image, the user U1 and the user U1's mother U10 are photographed in the image P72 determined as the first image, and the user U1 and the user U1's mother U10 are photographed in the image P83 determined as the second image, so the first person and the second person are determined to be the same person by the CPU 11. Therefore, the image P72 determined as the first image and the image P83 determined as the second image are not determined by the CPU 11 as an image combination.
[0071] refer to Figure 8 , if the CPU 11 or the user U1 determines the first image as the image P72 taken when the user U1 entered the elementary school and the CPU 11 determines the image P94 as the second image, the user U1 and the mother U10 of the user U1 are photographed in the image P72 determined as the first image, and the user U1 is photographed in the image P94 determined as the second image, so the first person and the second person are not determined to be the same person by the CPU 11. If it is assumed that the age of the user U1 included in the image P72 determined as the first image (the first age, the age when the image P72 was taken) is 6 years old and the age of the mother U10 of the user U1 (the first age, the age when the image P72 was taken) is 32 years old, and the age of the user U1 included in the image P94 determined as the second image (the second age, the age when the image P94 was taken) is 20 years old, the age difference is not within the threshold, so the image P72 and the image P94 are not determined by the CPU 11 as a combination of the first image and the second image.
[0072] If we assume that Figure 8 As shown in FIG. 1 , the CPU 11 or the user U1 determines the first image as the image P72 taken when the user U1 enters the elementary school. Figure 7As shown, when the CPU 11 determines the image P12 as the second image, the user U1 and the user U1's mother U10 are photographed in the image P72 determined as the first image, and the user U1's father U11 and the user U1's grandmother U21 are photographed in the image P12 determined as the second image, so the first person and the second person are not determined to be the same person by the CPU 11. If it is assumed that the age of the user U1 included in the image P72 determined as the first image (the first age, the age when the image P72 was taken) is 6 years old and the age of the user U1's mother U10 (the first age, the age when the image P72 was taken) is 32 years old, and the age of the user U1's father U11 included in the image P12 determined as the second image (the second age, the age when the image P94 was taken) is 6 years old, regardless of the age of the grandmother U21, the age difference is within the threshold, and the image P72 and the image P12 are determined by the CPU 11 as a combination of the first image and the second image.
[0073] If the storage server 10 is configured to store the first person and the second person as the same person (in Figure 5 38) or the difference between the first age and the second age is not within the threshold range (in Figure 5 If "No" is indicated in step 37 of the embodiment, the processing continues until at least one image combination is found, then a new second image (in Figure 6 "Yes" in step 41, step 42), and continue processing under the control of CPU11 (an example of a control mechanism).
[0074] If the storage server 10 is set to end the processing when at least one image combination is found, the CPU 11 controls the storage server 10 to end the processing when the combination of the first image and the second image is found (in Figure 6 If the storage server 10 is set to continue processing even if one image combination is found, the storage server 10 continues processing even if a combination of the first image and the second image is found (in Figure 6 41 (“No”).
[0075] When there is an undetermined second image (in Figure 6 In step 42 of "Yes", the CPU 11 determines the next second image ( Figure 5 Step 35), as described above, is controlled by CPU 11 to be repeated Figure 5 Step 36 to Figure 6 If the storage server 10 is configured to process the second image when there is no undetermined second image (in Figure 6 In step 42 of "No"), the first image remains undetermined (in Figure 6In step 43 of , “Yes” is given), and even if at least one image combination is found, the processing is not terminated and continues (in Figure 6 If "No" is obtained in step 44 of the method, the CPU 11 determines the next first image ( Figure 4 Step 32), and the CPU 11 controls to repeatedly perform Figure 4 The processing of step 33 is performed.
[0076] If it is set to end the processing of the storage server 10 when at least one image combination is detected (in Figure 6 41 in step 41), or there is no undetermined first image (in step 41 in ... Figure 6 43 in step 43), or the storage server 10 is set to end the processing even if an undetermined first image remains (in step 44). Figure 6 If "Yes" is obtained in step 44 of Figures 4 to 6 Processing shown.
[0077] In the above embodiment, if at least one image combination is detected, the processing of the storage server 10 is terminated (in Figure 6 However, the processing from step 32 may be performed without ending and the first image to be determined may be changed. The image combination of the first image and the second image is determined one by one for each first image.
[0078] Fig. 9 This is an example of a printed material printed by the printer server 20 .
[0079] As described above, it is assumed that the image P12 and the image P72 are determined as the image combination of the first image and the second image. The image file representing the first image, the image file representing the second image, and the print command are sent to the printer server 20 under the control of the CPU 11 (an example of a printer control unit). The printed matter 50 is printed in the printer server 20. The image P72 determined as the first image is printed in the left area 51 of the printed matter 50, and the image P12 determined as the second image is printed in the right area 52 of the printed matter 50.
[0080] Image P72 is an image taken when the user U1 was 6 years old, and is an image of the user U1's elementary school entrance ceremony. Image P12 is an image taken when the user U1's father U11 was 6 years old, and is an image of the user U1's father U11's elementary school entrance ceremony.
[0081] The printed matter 50 in which the images P12 and P72 of the mother and the child photographed at the same age are printed adjacent to each other can be obtained by the printer server 20. The printed matter 50 is mailed to the user U1, for example.
[0082] exist Fig. 9 In the example shown, the printed matter 50 has images P12 and P72 printed adjacent to each other. However, the CPU 11 (an example of a display control unit) of the storage server 10 may generate an image file representing the images P12 and P72 (or an image file representing the images P12 and P72 as one adjacent image) and an instruction to display the images P12 and P72 adjacent to each other on the display screen of the display device included in the client computer 1, and transmit the image file and instruction from the storage server 10 to the client computer 1. The images P12 and P72 are displayed adjacent to each other on the display screen of the display device included in the client computer 1. The CPU 11 (an example of a display control unit) may control the display of P12 and P72 adjacent to each other on the display screen of the display device 19A of the storage server 10.
[0083] According to this embodiment, images of different people taken at the same age can be found. In the above embodiment, the image of the user U1 and the image of the user U1's father U11 taken at the same age as the user when the image was taken (as long as the age difference is within the threshold) are found, but in addition to the image of the father and the image of the child taken at the same age, it is also possible to find images of brothers taken at the same age.
[0084] Furthermore, in the above-mentioned embodiment, all images stored in the hard disk 18 may be images of relatives or in-laws of the user. The first image and the second image to be detected are only images of relatives or in-laws such as mother and child, brothers, sisters, cousins, etc.
[0085] [Modifications]
[0086] Fig.10 and Fig.11 This is a flowchart showing the flow of processing for determining the second image. Fig.10 and Fig.11 The processing shown corresponds to Figure 5 The process of step 35 is performed.
[0087] exist Fig.10 and Fig.11 In the processing shown, the CPU 11 determines, as the second image, an image whose shooting date is more than a threshold value away from the shooting date of the first image.
[0088] The determined shooting date of the first image is read from the header of the image file representing the first image by the CPU 11 (step 61). When the shooting date of the first image cannot be read ("No" in step 62), the shooting date of the first image is estimated by the CPU 11 (step 63). The shooting date may also be estimated using AI, and when the first image includes a subject whose shooting date can be estimated, the shooting date is estimated using the subject.
[0089] The CPU 11 determines the second image candidate from the images other than the first image, and reads the shooting date of the second image candidate from the header of the image file representing the second image (step 64). When the shooting date of the second image candidate cannot be read ("No" in step 65), the CPU 11 estimates the shooting date of the second image candidate (step 66). Regarding the second image candidate, the shooting date can also be estimated by AI. When the second image candidate includes a subject whose shooting date can be estimated, the shooting date is estimated by using the subject.
[0090] If the shooting date of the first image is more than a threshold value (for example, 100 days, but other values are also possible) apart from the shooting date of the second image candidate ("Yes" in step 67), the second image candidate is determined as the second image by the CPU 11 (step 68). If the shooting date of the first image is not more than a threshold value apart from the shooting date of the second image candidate ("No" in step 67), another second image candidate is determined by the CPU 11 and the shooting year, month and day are read (step 64). The determination of the second image candidate can be performed in any manner, and can be performed in the order of shooting or randomly.
[0091] Fig.12 is a flowchart showing a partial processing flow of the second image determination processing, corresponding to Fig.11 The processing flow. Fig.10 The processing flow shown is implemented Fig.12 The processing flow shown.
[0092] exist Fig.12 In step 71, the CPU 11 determines whether the shooting year of the first image is different from or the same as the shooting year of the second image candidate (step 71). If they are different (“Yes” in step 71), the CPU 11 determines whether the shooting time of the first image is the same as or different from the shooting time of the second image candidate (step 72). If the shooting time is the same (“Yes” in step 72), the CPU 11 determines the second image candidate as the second image (step 73).
[0093] It is possible to find the first image and the second image that were taken at an event that took place at the same time, such as an entrance ceremony. As described above, it is possible to find images of the entrance ceremony or graduation ceremony of parents and images of the entrance ceremony or graduation ceremony of children, images of the entrance ceremony or graduation ceremony of an older brother and images of the entrance ceremony or graduation ceremony of a younger brother, etc. The same time period refers to a period that is generally felt to be the same, such as the same season, the same month, etc.
[0094] Fig.13 This is a flowchart showing the flow of processing for determining the second image. Fig.13 The processing shown also corresponds to Figure 5 The process of step 35 is as follows.
[0095] exist Fig.13 In the processing flow shown, an image having the same composition as that of the first image is used as the second image.
[0096] The composition of the first image determined by the CPU 11 is detected by the CPU 11 (step 81). A plurality of composition samples may be stored in advance and the composition detection may be performed by matching with the composition samples, or the position, type, size, etc. of the subject included in the image may be detected and the composition detection may be performed based on the position, type, size, etc. of the subject detected. In particular, the composition may be detected with emphasis on the position, size, etc. of the person. Subsequently, the second image candidate is determined by the CPU 11, and the composition of the determined second image candidate is also detected by the CPU 11 (step 82).
[0097] The CPU 11 compares the composition of the first image with the composition of the second image candidate. If the two compositions are substantially the same (the degree of composition consistency is above the threshold) ("Yes" in step 83), the CPU 11 determines the second image candidate as the second image (step 84). If the composition of the first image is not substantially the same as the composition of the second image candidate ("No" in step 83), the CPU 11 determines a new second image candidate, and the CPU 11 detects the composition of the new second image candidate (step 82). The composition of the first image is compared with that of the new second image candidate.
[0098] It is possible to find images of different people taken in the same age group with the same composition.
[0099] Fig.14 and Fig.15 This is also a flowchart showing the flow of processing for determining the second image. Fig.14 and Fig.15 The processing shown also corresponds to Figure 5 The process of step 35 is as follows.
[0100] Fig.14 and Fig.15The processing shown is a processing of using an image of the same shooting location as that of the first image as the second image.
[0101] The shooting location of the first image {GPS (Global Positioning System) data indicating the shooting location, etc.} is read from the header of the image file indicating the first image by the CPU 11 (step 91). When the shooting location of the first image cannot be read from the header of the image file indicating the first image by the CPU 11 due to the fact that the shooting location is not recorded in the header ("No" in step 92), the shooting location of the first image is estimated by the CPU 11 (step 93). For example, if an address display or a subject that is a feature indicating a location is captured in the first image, the shooting location is estimated based on these subjects by the CPU 11.
[0102] The second image candidate is determined by the CPU 11, and the shooting location of the second image candidate is read from the header of the image file representing the second image candidate by the CPU 11 (step 95). If the shooting location of the second image candidate cannot be read from the header of the image file representing the second image candidate by the CPU 11 due to the fact that the shooting location is not recorded in the header or the like ("No" in step 95), the shooting location of the second image candidate is estimated by the CPU 11 (step 96).
[0103] If the shooting location of the first image is the same as the shooting location of the second image candidate (it can be regarded as the same) ("Yes" in step 97), the second image candidate is determined as the second image by the CPU 11 (step 98). If the shooting location of the first image is different from the shooting location of the second image candidate ("No" in step 97), a new second image candidate is determined by the CPU 11, and the shooting location of the determined new second image candidate is read or estimated (steps 94, 96). The shooting location of the first image is compared with the shooting location of the new second image candidate (step 97).
[0104] It is possible to find images of different people of the same age group that were taken at the same location. For example, if a mother and a child went to the same school, it is possible to find images of the child's entrance ceremony, graduation ceremony, etc. in the same school and images of the child's parents' entrance ceremony, graduation ceremony, etc. in the same school.
[0105] Fig.16 This is also a flowchart showing the flow of processing for determining the second image. Fig.16 The processing shown also corresponds to Figure 5 The process of step 35 is as follows.
[0106] Fig.16The processing shown is a processing of specifying, as the second image, an image including at least one of the relatives or in-laws of the person included in the first image.
[0107] The CPU 11 reads information about relatives and in-laws of the first image from the header of the image file representing the first image (step 101). The CPU 11 determines a second image candidate, and reads information about relatives and in-laws of the person included in the second image candidate from the header of the image file representing the second image candidate (step 102).
[0108] CPU11 confirms whether the person included in the first image and the person included in the second image candidate represent at least one of the same relative or the same in-law based on the information about the relative or the in-law obtained from the first image and the information about the relative or the in-law obtained from the second image candidate (step 103). If they represent at least one of the same relative or the same in-law ("yes" in step 103), the second image candidate is determined as the second image by CPU11 (step 104). If they are neither the same relative nor the same in-law ("no" in step 103), a new second image candidate is determined by CPU11, and for the new second image candidate, CPU11 confirms whether the person included in the first image and the person included in the second image candidate represent at least one of the same relative or the same in-law (step 103).
[0109] In the above embodiment, CPU 11 uses the information about relatives and in-laws recorded in the header of the image file to confirm whether the person included in the first image and the person included in the second image candidate represent the same relative or at least one of the same in-laws. However, data representing facial images of relatives and in-laws of a certain user may be pre-stored in the hard disk 18 of the storage server 10 corresponding to the user, and the data representing the facial images may be used and AI may be used to determine whether the person included in the first image and the person included in the second image candidate are at least one of the relatives or in-laws.
[0110] It is possible to find images of relatives and in-laws taken in the same age group.
[0111] Fig.17 and Fig.18 This is also a flowchart showing the flow of processing for determining the second image. Fig.17 and Fig.18 The processing shown also corresponds to Figure 5 The process of step 35 is performed.
[0112] Fig.17 and Fig.18The processing shown is a processing of specifying an image of the same event as the event represented by the first image as the second image.
[0113] The event represented by the first image is read from the header of the image file representing the first image by the CPU 11 (step 111). When the event represented by the first image cannot be read from the header of the image file representing the first image by the CPU 11 due to the fact that the event is not recorded in the header ("No" in step 112), the event of the first image is estimated by the CPU 11 (step 113). For example, if a subject representing an event, an event name, etc. are captured in the first image, the event is estimated based on the subject and the event name by the CPU 11.
[0114] The second image candidate is determined by the CPU 11, and the event represented by the second image candidate is read from the header of the image file representing the second image candidate by the CPU 11 (step 114). When the event represented by the second image candidate cannot be read from the header of the image file representing the second image candidate by the CPU 11 due to the fact that the event is not recorded in the header or the like ("No" in step 115), the event represented by the second image candidate is estimated by the CPU 11 (step 116).
[0115] If the event represented by the first image is the same as the event represented by the second image candidate (they can be regarded as the same) ("Yes" in step 117), the second image candidate is determined as the second image by CPU 11 (step 118). If the event represented by the first image is different from the event represented by the second image candidate ("No" in step 117), a new second image candidate is determined by CPU 11, and the event represented by the determined new second image candidate is read or estimated (steps 114, 116). The event represented by the first image is compared with the event represented by the new second image candidate (step 117).
[0116] It is possible to find images of different people photographed in the same age group that depict the same event.
[0117] Fig.19 This is a flowchart showing the flow of processing for determining an image combination of a first image and a second image. Fig.19 The processing flow shown corresponds to Figure 5 The processing flow of step 40 is as follows.
[0118] Fig.19The processing shown is a processing for determining the first image and the second image as an image combination when the first person included in the first image is similar to the second person included in the second image and the first image and the second image are similar. The first image and the second image may be determined as an image combination when the first image and the second image are not similar but the first person included in the first image and the second person included in the second image are similar, or the first image and the second image may be determined as an image combination when the first person included in the first image and the second person included in the second image are not similar but the first image and the second image are similar.
[0119] The CPU 11 detects the first person from the first image and detects the first person from the determined second image ( Figure 5 The second person is detected in step 35, and the first similarity indicating the similarity between the first person and the second person is calculated by CPU 11 (step 121). If the calculated first similarity is less than the first threshold value ("No" in step 122), it is considered that the first person and the second person are not similar (for example, they are not relatives but other people), and another second image is re-determined by CPU 11.
[0120] If the calculated first similarity is greater than or equal to the first threshold value ("Yes" in step 122), the first person and the second person are considered to be similar (e.g., relatives), and the second similarity indicating the similarity between the first image and the second image is calculated by the CPU 11 (step 123). If the calculated second similarity is less than the second threshold value ("No" in step 124), the first similarity and the second similarity are considered to be dissimilar, and another second image is newly determined by the CPU 11.
[0121] If the calculated second similarity is equal to or greater than the second threshold value (YES in step 124 ), the first similarity and the second similarity are considered to be similar, and the CPU 11 determines an image combination of the first image and the second image (step 125 ).
[0122] An image combination of a first image and a second image including a second person similar to the first person, which are images of different persons photographed in the same age group, can be determined, and the image combination is also similar to the first image and the second image.
[0123] In the above-described embodiment, it is assumed that the manager of the image processing system obtains the user's consent regarding the image analysis of the user.
[0124] The processing unit that performs the above-mentioned processing includes, in addition to the CPU 11 that executes software and functions as various processing units, programmable logic devices such as FPGA (field-programmable gate array) that can change the circuit structure after manufacturing, and ASIC (application specific integrated circuit) and other processors with circuit structures specially designed for performing specific processing, that is, dedicated circuits.
[0125] A processing unit may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, CPUs and FPGAs). As an example of a plurality of processing units being composed of one processor, first, as represented by computers such as client computers or servers, there is the following form: a processor is composed of a combination of one or more CPUs and software, and the processor functions as a plurality of processing units. Second, as represented by a system on a chip, there is the following form: a processor that implements the functions of the entire system including a plurality of processing units using an IC (integrated circuit) chip. In this way, various processing units are composed of one or more various processors as a hardware structure.
[0126] Furthermore, more specifically, the hardware structure of these various processors is a circuit formed by combining circuit elements such as semiconductor elements.
[0127] Explanation of symbols
[0128] 1-n-computer, 10-storage server, 11-CPU, 12-communication device, 13-memory card reader / writer, 14-memory card, 15-micro optical drive, 16-micro optical disk, 17-hard disk drive, 18-hard disk, 19-memory, 19A-display device, 20-printer server, 50-printed material, 51-left area of the printed material, 52-right area of the printed material, P1-P300-image, U1-U3-user, U10-mother, U11-father, U21-grandmother, U31-child.
Claims
1. An image processing device comprising: a reading mechanism for reading an image from a recording device recording a plurality of images accessible to a user; A first image determining unit determines any one of the plurality of images as a first image; A first person recognition unit recognizes a first person captured in the first image; a first age estimating means for estimating the age of the first person at the shooting date and time of the first image; a second image determining means for determining any one image other than the first image from the plurality of images as a second image; A second person recognition unit recognizes a second person captured in the second image; a second age estimating means for estimating the age of the second person at the shooting date and time of the second image; A person determination unit determines whether the first person identified and the second person identified are the same person; an age difference determination unit for determining whether the difference between the estimated age of the first person and the estimated age of the second person is within a threshold range; and The image combination detection means detects the first image and the second image as an image combination when the person determination means determines that the first person and the second person are not the same person and the age difference determination means determines that the age difference between the first person and the second person is within a threshold value.
2. The image processing device according to claim 1, wherein: The second image specifying means specifies, as the second image, an image whose shooting date is separated from the shooting date of the first image by a threshold or more.
3. The image processing device according to claim 1 or 2, wherein: The second image specifying means specifies, as the second image, an image that was captured in a year different from the year in which the first image was captured and in the same period as the period in which the first image was captured.
4. The image processing device according to claim 1 or 2, wherein: The second image specifying means specifies an image having the same composition as that of the first image as the second image.
5. The image processing device according to claim 1 or 2, wherein: The second image specifying means specifies, as the second image, an image captured at the same location as that at which the first image was captured.
6. The image processing device according to claim 1 or 2, wherein: The second image specifying means specifies, as the second image, an image including a relative or in-law of the person included in the first image.
7. The image processing device according to claim 1 or 2, wherein: The second image specifying means specifies, as the second image, an image captured in the same event as the event in which the first image was captured.
8. The image processing device according to claim 1 or 2, The image processing device further comprises: The control means controls the image processing device to repeat at least one of the determination of the first image or the determination of the second image until at least one of the image combination is detected.
9. The image processing device according to claim 1 or 2, The image processing device further comprises: a similarity calculation unit that calculates at least one of a first similarity between the first person and the second person or a second similarity between the first image and the second image, The image combination detection unit detects the first image and the second image as an image combination when at least one of the first similarity and the second similarity is equal to or greater than a threshold value.
10. The image processing device according to claim 1 or 2, wherein: A plurality of images of relatives or in-laws of a specific person are recorded in the recording device.
11. The image processing device according to claim 1 or 2, The image processing device further comprises: The display control means controls the display device so as to display the first image and the second image adjacent to each other.
12. The image processing device according to claim 1 or 2, The image processing device further comprises: The printer control mechanism controls the printer so as to print the first image and the second image adjacent to each other.
13. The image processing device according to claim 1 or 2, wherein: The first image specifying means specifies the first image based on an instruction from a user.
14. An image processing method, wherein: The reading mechanism reads an image from a recording device recording a plurality of images accessible to a user. The first image determining means determines any one of the plurality of images as the first image. The first person recognition unit recognizes a first person captured in the first image. The first age estimating means estimates the age of the first person at the shooting date and time of the first image. The second image determining means determines any one image other than the first image from the plurality of images as the second image, The second person recognition unit recognizes a second person captured in the second image. The second age estimating means estimates the age of the second person at the shooting date and time of the second image. The person determination unit determines whether the first person identified and the second person identified are the same person. The age difference determination unit determines whether the difference between the estimated age of the first person and the estimated age of the second person is within a threshold range. When the person determination unit determines that the first person and the second person are not the same person and the age difference determination unit determines that the age difference between the first person and the second person is within a threshold range, the image combination detection unit detects the first image and the second image as an image combination.
15. A recording medium storing a computer-readable program, the program being a program for controlling a computer of an image processing device, the program controlling the computer of the image processing device to: reading an image from a recording device recording a plurality of images accessible to a user, Determine any one of the plurality of images as the first image, identifying a first person captured in the first image, estimating the age of the first person at the shooting date and time of the first image, determining any one image other than the first image from the plurality of images as the second image, identifying a second person captured in the second image, estimating the age of the second person based on the shooting date and time of the second image, determining whether the first person identified and the second person identified are the same person, determining whether the difference between the estimated age of the first person and the estimated age of the second person is within a threshold range, When it is determined that the first person and the second person are not the same person and when it is determined that the age difference between the first person and the second person is within a threshold range, the first image and the second image are detected as an image combination.
