Information processing device, information processing method, and program

Abstract

To perform face analysis in a more reliable manner.SOLUTION: A recognition unit recognizes a face in a photographed image in which a person is captured, and an analysis unit performs face analysis using a successfully recognized face as an analysis target. If there is a face that fails in being recognized, the analysis unit analyzes a face included in a designation area designated by a user in the photographed image as the analysis target. The present technology can be applied to a recommendation system for presenting recommended makeup to users.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present technology relates to an information processing apparatus, an information processing method, and a program, and more particularly to an information processing apparatus, an information processing method, and a program that enable more reliable face analysis.

Background Art

[0002] Patent Document 1 discloses a cosmetic vending machine that performs skin diagnosis based on data of a user's face photographed by a camera and displays, on a screen as information for the user's product selection, cosmetics (for example, foundation) most suitable for the skin.

[0003] On the other hand, in recent years, face diagnosis applications (apps) have been provided that perform face diagnosis such as processing a face image into a state where makeup expected by the user is applied or outputting advice by using a user's face image or having the user answer questions.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the above-described face diagnosis app, face recognition processing is performed on an image of the user's own face photographed with a smartphone, and face analysis is performed on the face for which recognition has succeeded as an analysis target. When face recognition fails, it is notified that face analysis cannot be performed, and the user repeats photographing their own face until face recognition succeeds.

[0006] However, in an image photographed in the past, a face for which recognition fails cannot be set as an analysis target.

[0007] This technology was developed in light of these circumstances and aims to enable more reliable facial analysis. [Means for solving the problem]

[0008] The information processing device of this technology comprises a recognition unit that recognizes faces in a captured image of a person, and an analysis unit that performs face analysis on faces that have been successfully recognized. If there are faces that have not been recognized, the analysis unit analyzes faces that are included in a designated area specified by the user in the captured image.

[0009] The information processing method of this technology involves an information processing device recognizing faces in a captured image containing a person, performing face analysis on faces that were successfully recognized, and if there are faces that were not recognized, analyzing the faces that are included in a designated area specified by the user in the captured image.

[0010] The program for this technology causes a computer to recognize faces in a captured image containing a person, perform face analysis on faces that have been successfully recognized, and if there are faces that have not been recognized, to perform analysis on faces that are included in a specified area specified by the user in the captured image.

[0011] In this technology, faces are recognized in captured images containing people, and face analysis is performed on faces that are successfully recognized. If there are faces that are not successfully recognized, the faces that fall within a specified area designated by the user in the captured image are used for analysis. [Effects of the Invention]

[0012] This technology makes it possible to perform facial analysis more reliably.

[0013] The effects described herein are not necessarily limited to those described herein and may include any of the effects described herein. [Brief explanation of the drawing]

[0014] [Figure 1] It is a diagram showing a configuration example of a recommendation system. [Figure 2] It is a block diagram showing a hardware configuration example of a photo creation game machine. [Figure 3] It is a block diagram showing a hardware configuration example of a server. [Figure 4] It is a block diagram showing a hardware configuration example of a mobile terminal. [Figure 5] It is a diagram for explaining the flow of presenting recommendation information based on a face image. [Figure 6] It is a block diagram showing a functional configuration example of a recommendation system. [Figure 7] It is a diagram showing an example of user data. [Figure 8] It is a diagram for explaining the flow of operation of a recommendation system. [Figure 9] It is a diagram showing an example of a screen of a recommended site displayed on a mobile terminal. [Figure 10] It is a diagram showing an example of a screen of a recommended site displayed on a mobile terminal. [Figure 11] It is a diagram showing an example of a screen of a recommended site displayed on a mobile terminal. [Figure 12] It is a diagram showing an example of a screen of a recommended site displayed on a mobile terminal. [Figure 13] It is a diagram showing an example of a screen of a recommended site displayed on a mobile terminal. [Figure 14] It is a diagram showing an example of a screen of a recommended site displayed on a mobile terminal. [Figure 15] It is a diagram showing an example of a screen of a recommended site displayed on a mobile terminal. [Figure 16] It is a flowchart for explaining the flow of analysis target setting processing. [Figure 17] It is a diagram for explaining an application example of the present technology.

Modes for Carrying Out the Invention

[0015] The following describes the embodiments for implementing this technology. The description will be given in the following order.

[0016] 1. Configuration of the recommendation system 2. Flowchart for presenting recommendation information based on facial images 3. Examples of screen transitions on mobile devices 4. Flow of the process for setting the analysis target 5. Variations

[0017] <1. Configuration of the recommendation system> (Overview of the recommendation system) Figure 1 shows an overview of the recommendation system according to an embodiment of this technology.

[0018] The recommendation system 1 in Figure 1 consists of a photo creation game machine 10, an image management server 20, a recommendation server 30, and mobile terminals 40-1 and 40-2.

[0019] Each device constituting the recommended system 1 is connected to a network NW, either directly or indirectly, and communicates with one another. The network NW includes the internet and local area networks, and may consist of wired communication lines or wireless communication lines.

[0020] Furthermore, in recommendation system 1, the image management server 20 and the recommendation server 30 are configured separately, but they may also be configured as a single unit.

[0021] The photo creation game machine 10 is installed in stores and amusement facilities such as arcades, and provides users with a photo creation game by accepting coins equivalent to the play fee. Specifically, the photo creation game machine 10 accepts editing operations (doodling) on ​​the captured image obtained by taking a picture of the user, and provides the captured image to the user by printing it on sticker paper.

[0022] The users of the photo creation game machine 10 are mainly high school girls and young women. The photo creation game machine 10 can be enjoyed by two users per group. Of course, it can also be enjoyed by three or more users, or by a single user.

[0023] Although Figure 1 shows only one photo creation game machine 10, in the recommended system 1, there are actually multiple photo creation game machines 10.

[0024] The image management server 20 manages the images captured by the photo creation game machine 10, and also manages information about users who have become members of the website (image acquisition site) operated by the manufacturer that produces and sells the photo creation game machine 10.

[0025] Users who become members will be able to view the images taken using the photo creation game machine 10 on their mobile devices 40-1 and 40-2.

[0026] The image management server 20 retrieves images taken by the photo creation game machine 10 (such as at a store where the photo creation game machine 10 is installed) for each member user. The images taken by each user may be supplied to the recommendation server 30 as needed.

[0027] The recommendation server 30 uses user-specific images from the image management server 20 to provide a website (recommendation site) that recommends products and services to each user. Specifically, the recommendation server 30 generates recommendation information for recommending products and services based on the analysis results obtained by analyzing each user's images, and provides it to the mobile terminals 40-1 and 40-2 owned by each user.

[0028] Mobile terminals 40-1 and 40-2 consist of mobile phones (smartphones), mobile PCs (personal computers), tablet devices, etc., owned by each user. In the following, when mobile terminals 40-1 and 40-2 are not distinguished, they will simply be referred to as mobile terminal 40. Although only two mobile terminals 40 are shown in Figure 1, in reality, multiple mobile terminals 40 exist in Recommendation System 1.

[0029] (Hardware configuration of the photo creation game machine) Figure 2 is a block diagram showing an example of the hardware configuration of the photo creation game machine 10.

[0030] As shown in Figure 2, the photo creation game machine 10 is configured such that a pre-selection unit 102, a shooting unit 103, an editing unit 104, and a printing unit 105 are connected to the PC unit 101 via an external input / output interface 106.

[0031] The CPU (Central Processing Unit) 111, ROM (Read Only Memory) 112, and RAM (Random Access Memory) 113 that make up the PC unit 101 are interconnected by a bus 114.

[0032] The CPU 111 executes a predetermined program and controls the overall operation of the photo creation game machine 10. The ROM 112 stores the programs and data executed by the CPU 111. The RAM 113 temporarily stores the data and programs processed by the CPU 111.

[0033] An input / output interface 115 is further connected to the bus 114. A storage unit 116, a communication unit 117, and a drive 118 are connected to the input / output interface 115.

[0034] The memory unit 116 is a non-volatile storage medium such as a hard disk or flash memory. The memory unit 116 stores various setting information supplied by the CPU 111. The information stored in the memory unit 116 is read out by the CPU 111 as needed.

[0035] The communication unit 117 is an interface to a network such as the Internet. The communication unit 117 communicates with external devices according to the control of the CPU 111. The communication unit 117 transmits the image selected by the user to the image management server 20. The image transmitted from the communication unit 117 is allocated a predetermined storage area and stored, and is displayed on the mobile terminal 40 that accesses the image acquisition site or downloaded to the mobile terminal 40.

[0036] A removable media 119, such as an optical disc or semiconductor memory, is appropriately mounted in the drive 118. Programs and data read from the removable media 119 by the drive 118 are supplied to the CPU 111 and stored or installed in the storage unit 116.

[0037] An external input / output interface 106 is connected to the input / output interface 115. Control of each component by the PC unit 101 is performed via the external input / output interface 106.

[0038] The pre-selection unit 102 performs pre-selection processing for the user before shooting. The pre-selection unit 102 consists of a touch panel monitor 121, a speaker 122, and a coin processing unit 123.

[0039] The touch panel monitor 121 displays various selection screens according to the control of the PC unit 101 and accepts user input on the selection screens. Input signals representing the content of the user's input are supplied to the PC unit 101, and various settings are made.

[0040] Speaker 122 outputs voice guidance for various selection screens, background music (BGM), sound effects, and more.

[0041] The coin processing unit 123 detects when a coin is inserted into the coin slot. When the coin processing unit 123 detects that a predetermined amount of coins has been inserted, it outputs a start signal to the PC unit 101 to instruct the game to start.

[0042] The shooting unit 103 performs shooting processing targeting a user who is in the shooting space formed inside the casing of the photo creation game machine 10. The shooting unit 103 consists of a camera 131, a touch panel monitor 132, a lighting device 133, and a speaker 134.

[0043] The camera 131 takes pictures according to the shutter control by the PC unit 101 and outputs the image data obtained from the pictures to the PC unit 101.

[0044] The touch panel monitor 132 is located near the camera 131. The touch panel monitor 132 has the function of a live view monitor that displays moving images captured by the camera 131, and the function of displaying various GUIs and accepting user selection operations.

[0045] The lighting device 133 is a strobe unit installed in the shooting space and emits light according to the lighting control signal supplied from the PC unit 101.

[0046] Speaker 134 is installed at a predetermined location within the shooting space and outputs guidance voices for the shooting process, background music, sound effects, etc.

[0047] The editorial unit 104 performs post-production editing. The editorial unit 104 consists of a tablet-integrated monitor 141, stylus pens 142A and 142B, and speaker 143.

[0048] The tablet's built-in monitor 141 and the stylus pens 142A and 142B are provided on each of the two sides of the housing that constitutes the editorial unit 104. When there is no need to distinguish between the stylus pens 142A and 142B, they are simply referred to as stylus pen 142.

[0049] The tablet's built-in monitor 141 displays an editing screen according to the control of the PC unit 101 and accepts user input using a stylus pen 142 on the editing screen. Input signals representing the user's operations are supplied to the PC unit 101, and the captured image to be edited is performed.

[0050] Speaker 143 is built into part of the casing that makes up the editorial department 104 and outputs voice guidance for editing processes, background music, sound effects, etc.

[0051] The printing unit 105 is configured to include a printer 151. A sticker paper unit 152 is mounted on the printer 151.

[0052] The printer 151 prints an image onto the sticker paper stored in the sticker paper unit 152 based on the print data supplied from the PC unit 101. The printer 151 then ejects the sticker paper with the printed image to the sticker paper output port.

[0053] (Hardware configuration of image management server and recommendation server) Figure 3 is a block diagram showing examples of the hardware configuration of the server 200 that makes up the image management server 20 and the recommendation server 30, respectively.

[0054] In Figure 3, the CPU 211 executes various processes according to the program stored in the ROM 212 or the program loaded into the RAM 213. The RAM 213 also stores data necessary for the CPU 211 to execute these processes as needed.

[0055] The CPU 211, ROM 212, and RAM 213 are interconnected via a bus 214. An input / output interface 215 is also connected to this bus 214.

[0056] The input / output interface 215 is connected to an input unit 216, an output unit 217, a storage unit 218, and a communication unit 219.

[0057] The input unit 216 consists of a keyboard, mouse, etc. The output unit 217 consists of a display made of liquid crystal or organic EL (Electro-Luminescence), and speakers, etc. The storage unit 218 consists of a hard disk, etc. The communication unit 219 consists of a wired communication module for wired communication, a wireless communication module for wireless communication, etc.

[0058] The communications unit 219 performs communication processing via the network NW.

[0059] The input / output interface 215 is also connected to a drive 220 as needed, and removable media 221, such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, is appropriately mounted. Computer programs read from these are installed in the storage unit 218 as needed.

[0060] (Hardware configuration of mobile devices) Figure 4 illustrates an example of the hardware configuration of the mobile terminal 40.

[0061] In Figure 4, the CPU 411 executes various processes according to the program stored in the ROM 412 or the program loaded into the RAM 413. The RAM 413 also stores data necessary for the CPU 411 to execute these processes as appropriate.

[0062] The CPU 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output interface 415 is also connected to this bus 414.

[0063] The input / output interface 415 is connected to an input unit 416, an output unit 417, a storage unit 418, and a communication unit 419.

[0064] The input unit 416 consists of keys, buttons, a touch panel, a camera, and a microphone. The output unit 417 consists of a display made of liquid crystal or organic EL, which is formed together with the touch panel, and a speaker. The storage unit 418 consists of, for example, NAND flash memory. The communication unit 419 consists of a communication module that performs wireless communication.

[0065] <2. Flow of presenting recommendation information based on facial images> The following describes an example of presenting recommendation information to a user based on a facial image extracted from a captured image generated by the photo creation game machine 10.

[0066] (Flow of presenting recommendation information) Figure 5 illustrates the flow of presenting recommendation information based on facial images.

[0067] Figure 5 shows the photo creation game machine 10, recommendation server 30, and mobile terminal 40 that constitute the recommendation system 1.

[0068] Unlike the unstable shooting using a smartphone, the photo creation game machine 10 takes photos under consistent conditions using a user who owns a mobile device 40 as the subject, resulting in stable, high-quality images. The captured image P11 is supplied from the photo creation game machine 10 to the recommendation server 30 via the image management server 20.

[0069] The recommendation server 30 performs face recognition and analysis on the captured image P11. Specifically, the recommendation server 30 extracts face images by recognizing faces in the captured image P11 and performs face analysis on the extracted face images. Based on the analysis results obtained, the recommendation server 30 generates recommendation information to recommend products and services to the user. For example, based on the face image analysis results, recommendation information R31 is generated to recommend makeup and cosmetics that match the user's face. The recommendation server 30 then transmits the generated recommendation information R31 to the mobile terminal 40 owned by the user.

[0070] The photo creation game machine 10 may perform face recognition and analysis on the captured image P11, and supply the obtained analysis results, along with the captured image P11, to the recommendation server 30 via the image management server 20. In this case, the recommendation server 30 generates recommendation information R31 based on the analysis results from the photo creation game machine 10.

[0071] Alternatively, the photo creation game machine 10 may perform face recognition and analysis on the captured image P11, generate recommendation information R31 based on the analysis results, and supply the generated recommendation information R31, along with the captured image P11, to the recommendation server 30 via the image management server 20. In this case, the recommendation server 30 transmits the recommendation information R31 from the photo creation game machine 10 to the mobile terminal 40.

[0072] Furthermore, the mobile terminal 40 may acquire the captured image P11 from the recommendation server 30, perform face recognition and analysis on the captured image P11, and generate recommendation information R31 based on the obtained analysis results.

[0073] Following the process described above, recommendation information based on the user's facial image can be presented to the user on the mobile device 40.

[0074] (Example of a recommendation system's functional configuration) Figure 6 is a block diagram showing an example of the functional configuration of Recommendation System 1, which realizes the presentation of recommendation information based on facial images as described above.

[0075] Figure 6 shows the photo creation game machine 10, image management server 20, recommendation server 30, and mobile terminal 40 that constitute the recommendation system 1, but a detailed explanation of the functional configuration of the image management server 20 and mobile terminal 40 is omitted.

[0076] The photo creation game machine 10 includes the aforementioned camera 131, touch panel monitor 132, lighting device 133, and printer 151, as well as a control unit 510. Each functional block constituting the control unit 510 is realized by the CPU 111 executing a predetermined program.

[0077] The control unit 510 is configured to include an imaging processing unit 511, an image processing unit 512, and a printing processing unit 513.

[0078] The shooting processing unit 511 controls the camera 131, touch panel monitor 132, and lighting device 133 installed in the shooting space to perform shooting processing targeting the user.

[0079] The image processing unit 512 performs various image processing operations on multiple captured images.

[0080] For example, the image processing unit 512 applies predetermined image processing to the subject area of ​​the captured image. Specifically, the image processing unit 512 performs retouching processing to modify (enlarge / reduce, transform, change color, etc.) the contour of the user's face and facial features such as eyes, nose, mouth, and cheeks in the captured image. In addition, the image processing unit 512 applies editing processing to the captured image in accordance with the user's editing operations (doodling).

[0081] Editing is applied to images selected by the user, according to the user's editing operations, whereas retouching is basically applied uniformly to all images, regardless of user operation. As a result of these image processing steps, a processed image is generated, separate from the unprocessed image.

[0082] The printing processing unit 513 controls the printer 151 to print the processed image, which has undergone image processing, onto sticker paper and output it. The processed image printed on the sticker paper may be edited according to the user's editing operations.

[0083] Furthermore, the processed images, along with the unprocessed captured images, are sent from the photo creation game machine 10 to the image management server 20 and managed on a per-user basis.

[0084] The recommendation server 30 comprises a control unit 530 and a user database 550. Each functional block constituting the control unit 530 and the user database 550 are realized by the CPU 211 executing a predetermined program.

[0085] The control unit 530 is configured to include a web page provisioning unit 531, a face recognition unit 532, a face analysis unit 533, a recommendation information generation unit 534, and a data management unit 535.

[0086] The web page provision unit 531 creates web pages that constitute a website (recommended site) to be viewed on the mobile terminal 40 and provides them to the mobile terminal 40.

[0087] The face recognition unit 532 recognizes a face in a captured image when it receives an image from the image management server 20 of a user corresponding to a mobile terminal 40 that has accessed a recommended site.

[0088] The face analysis unit 533 performs face analysis on face images of faces that have been successfully recognized by the face recognition unit 532 in the captured image.

[0089] Face recognition by the face recognition unit 532 and analysis by the face analysis unit 533 are performed on captured images that have not undergone image processing (original images) from among the captured images managed by the image management server 20. Hereinafter, captured images that have not undergone image processing will be referred to as pre-processing images, and among the pre-processing images, the captured images that are the target of face recognition and analysis will be referred to as diagnostic images.

[0090] The recommendation information generation unit 534 generates recommendation information for the user corresponding to the face image based on the analysis results obtained by the face analysis unit 533 performing analysis on the face (face image) that was successfully recognized.

[0091] The data management unit 535 manages user data for users who are registered on the recommendation site and whose mobile devices 40 have accessed the recommendation site. The user data is stored in the user database 550.

[0092] Figure 7 shows an example of user data.

[0093] As shown in Figure 7, the user data d550 stored in the user DB550 is configured by associating a user ID with a face image ID and one or more past recommendation information.

[0094] The User ID is unique identification information for users registered on the recommendation site. The Face Image ID is unique identification information for face images analyzed for users identified by the User ID. Past recommendation information is recommendation information generated for users identified by the User ID. Past recommendation information is added and accumulated, for example, each time an analysis is performed on a face image identified by the Face Image ID.

[0095] With user data d550 having this structure, users registered on the recommendation site can view previously presented recommendation information.

[0096] (How the recommendation system works) Referring to Figure 8, the operation flow of the recommendation system will be explained. The process in Figure 8 begins when the photo creation game machine 10 starts providing a game to the user.

[0097] First, in step S11, the photo creation game machine 10 (shooting processing unit 511) performs a shooting process. In the shooting process, multiple shots are taken with the user as the subject, generating multiple captured images.

[0098] During the shooting process, once multiple captured images (pre-processing images) are generated, the process proceeds to step S12, where the image processing unit 512 performs image processing on the captured images.

[0099] As described above, multiple captured images are retouched automatically without user intervention, and also edited according to user editing operations. As a result, multiple processed images are generated in addition to the multiple unprocessed images before any image processing was performed.

[0100] In step S13, the printing processing unit 513 controls the printer 151 to print at least one of the multiple processed images that have undergone image processing onto sticker paper and output it.

[0101] Subsequently, in step S14, the communication unit 117 (Figure 2) of the photo creation game machine 10 transmits multiple pre-processed and post-processed images generated in one game to the image management server 20.

[0102] The processed images sent to the image management server 20 are displayed on the mobile device 40 of users who have become members of the image acquisition site, or downloaded to the mobile device 40.

[0103] Here, members registered on the image acquisition site provided by the image management server 20 and members registered on the recommendation site provided by the recommendation server 30 are linked by a predetermined method. For example, a user can register themselves as the same member on both the image acquisition site and the recommendation site by using the email address of their mobile device 40 or their SNS (Social Networking Service) account as a common user ID.

[0104] In this state, for example, when a user accesses a recommended site from a mobile terminal 40 and performs an operation on the login screen that is displayed, in step S41, the communication unit 419 of the mobile terminal 40 sends a login request to the recommendation server 30 to log in to the recommended site.

[0105] In step S31, when the web page provider unit 531 of the recommendation server 30 accepts a login request from the mobile terminal 40, it retrieves (calls) the pre-processing image and post-processing image of the user of the mobile terminal 40 that sent the login request from the image management server 20.

[0106] In step S32, the web page provider unit 531 of the recommendation server 30 displays a selection screen on the mobile terminal 40 for selecting a diagnostic image to be used for face recognition and analysis, and a selection screen for selecting a face to be analyzed within the diagnostic image.

[0107] Since diagnostic images often depict multiple subjects, such as two people, a single diagnostic image can be associated with multiple users (members). In this case, even if a diagnostic image associated with a particular user can be identified, it is not possible to identify which subject in that diagnostic image represents that user.

[0108] Therefore, by having the user select themselves from the subjects in the diagnostic image, it becomes possible to identify which subject in the diagnostic image is the user. Specifically, on the selection screen, the user is asked to select one of the faces recognized in the diagnostic image. In the following, it is assumed that the diagnostic image contains two users.

[0109] In step S42, the communication unit 419 of the mobile terminal 40 transmits selection information (selected diagnostic images or faces) to the recommendation server 30 in response to the user's operation on the selection screen.

[0110] As the user repeatedly interacts with the selection screen, in step S33, the face recognition unit 532 of the recommendation server 30 recognizes a face in the diagnostic image selected by the user based on the selection information from the mobile terminal 40. The face analysis unit 533 identifies which face recognized in the diagnostic image belongs to the user of the mobile terminal 40 based on the selection information from the mobile terminal 40, and performs face analysis on that face image.

[0111] Next, in step S34, the web page provision unit 531 of the recommendation server 30 displays a question screen on the mobile terminal 40 that asks the user a question and allows them to select an answer to that question.

[0112] In step S44, the communication unit 419 of the mobile terminal 40 transmits answer information representing the user's selected answers to the questions to the recommendation server 30, in response to the user's operation on the question screen. The questions to the user may include, for example, questions about the user's skin color or questions about the makeup the user usually wears.

[0113] The display of such question screens and the submission of answer information are often performed multiple times.

[0114] Subsequently, in step S35, the recommendation information generation unit 534 of the recommendation server 30 generates recommendation information for the user of the mobile terminal 40 based on the analysis results obtained by the face analysis unit 533 performing face analysis on the diagnostic image and multiple response information from the mobile terminal 40. Here, for example, recommendation information is generated to recommend at least one of the makeup-related products and services based on the analysis results of the face image and the response information to the above-mentioned questions.

[0115] In step S36, the web page provision unit 531 of the recommendation server 30 sends a web page containing recommendation information to the mobile terminal 40.

[0116] Then, in step S44, the output unit 417 (Figure 4) of the mobile terminal 40 presents the user with a web page containing recommendation information from the recommendation server 30.

[0117] <3. Examples of screen transitions on mobile devices> Next, we will explain the screen transitions of the recommendation site viewed by the user using the mobile terminal 40 within the operation flow of the recommendation system 1 described above. Here, we will show an example of screen transitions up to the point where facial analysis of the user who is registered as a member of the recommendation site is performed (steps S41 and S42 in Figure 8).

[0118] First, when the mobile terminal 40 accesses the recommended site, the login screen 710 shown in Figure 9 is displayed on the display (output unit 417) of the mobile terminal 40. The login screen 710 is provided with a user icon display area 711, a password input box 712, and a login button 713.

[0119] The user icon display area 711 is an area where icons set for each user of the mobile terminal 40 are displayed. The icons displayed in the user icon display area 711 may be human-shaped figures automatically set by the system, as shown in Figure 9, or they may be user face images set by the user.

[0120] The password input box 712 is a text box for the user to enter a password to log in to the recommended site. When the correct password is entered in the password input box 712 and the user presses the login button 713, a login request is sent from the mobile terminal 40 to the recommendation server 30 (step S41 in Figure 8), and the login is completed.

[0121] Once login is complete, the mobile device 40's display shows the diagnostic start screen 720 shown in Figure 10. The diagnostic start screen 720 includes a terms of service display area 721, a checkbox 722, and a diagnostic start button 723.

[0122] The Terms of Use display area 721 is the area where the terms of use and privacy policy of the recommended site are displayed. The checkbox 722 is a GUI for the user to enter a checkmark indicating that they agree to the terms of use and privacy policy displayed in the Terms of Use display area 721. The Diagnosis Start button 723 is a button that is pressed to start the face diagnosis to generate recommendation information.

[0123] When the user presses the diagnostic start button 723 with a check mark in checkbox 722, the image selection screen 730 shown in Figure 11 is displayed on the mobile terminal 40's display as one of the selection screens described above. The image selection screen 730 is provided with a cautionary note display area 731 and an image selection area 732.

[0124] The cautionary information display area 731 is an area where cautionary information is displayed when selecting a diagnostic image in the image selection area 732. Specifically, the cautionary information display area 731 displays text encouraging the user to select an image as a diagnostic image in which the face is facing forward and the eyes are not closed.

[0125] The image selection area 732 displays multiple captured images that could be candidates for diagnostic images, for example, for each game provided by the photo creation game machine 10. In the example in Figure 11, five captured images from each of the two games recently played by a user logged into the recommendation site are displayed. The image selection area 732 displays the processed images from the pre-processing and post-processing images retrieved from the image management server 20.

[0126] When one of the captured images (processed image) is selected in the image selection area 732, the mobile terminal 40's display shows the face selection screen 740 shown in Figure 12 as another selection screen. The face selection screen 740 includes a diagnostic image display area 741, a diagnostic image re-selection button 742, and a cropping screen display button 743.

[0127] The diagnostic image display area 741 is the area where the diagnostic image selected in the image selection area 732 (Figure 11) is displayed. As mentioned above, the diagnostic image displayed in the diagnostic image display area 741 is a processed image that has undergone image processing. In other words, the face in the diagnostic image has been retouched, with its contours, eyes, nose, mouth, cheeks, etc., being modified or corrected.

[0128] Furthermore, in the diagnostic image displayed in the diagnostic image display area 741, a face frame is superimposed on the face that has been successfully recognized. In the example in Figure 12, the faces of two users in the diagnostic image are recognized, and face frames FF1 and FF2 are superimposed on each of their faces. If face frame FF1 is selected in the diagnostic image display area 741, the face (face image) of the user on the left of the two users in the diagnostic image is selected as the target of face analysis. On the other hand, if face frame FF2 is selected in the diagnostic image display area 741, the face (face image) of the user on the right of the two users in the diagnostic image is selected as the target of face analysis.

[0129] The diagnostic image re-selection button 742 is used to re-select a diagnostic image displayed in the diagnostic image display area 741. When the diagnostic image re-selection button 742 is pressed, the image selection screen 730 (Figure 11) is displayed again on the display of the mobile terminal 40.

[0130] The trimming screen display button 743 is pressed when, in the diagnostic image displayed in the diagnostic image display area 741, there are faces that are not superimposed with a face frame, that is, when there are faces in the diagnostic image that have failed to be recognized.

[0131] When the user presses the trimming screen display button 743, the display of the mobile terminal 40 shows the trimming screen (face area selection screen) 750 shown in Figure 13 as one of the selection screens. The trimming screen 750 is provided with a cautionary note display area 751, a diagnostic image display area 752, and an OK button 753.

[0132] The cautionary information display area 751 is an area in the diagnostic image display area 752 where cautionary information is displayed when the user of the mobile terminal 40 specifies (selects) the area of ​​their own face. Specifically, the cautionary information display area 751 displays text prompting the user to specify only the area of ​​their own face, including the top of their head, in the diagnostic image display area 752.

[0133] The diagnostic image display area 752 is the area where the diagnostic image selected in the image selection area 732 (Figure 11) is displayed. In the diagnostic image displayed in the diagnostic image display area 752, a rectangular cropping frame TF is displayed for specifying the face area. The user can specify the area of ​​their own face by changing the position and size of the cropping frame TF through operations on the mobile terminal 40.

[0134] When the OK button 753 is pressed in the diagnostic image display area 752 with the face area specified by the trimming frame TF, the specified face is selected as the target for face analysis.

[0135] If any of the faces successfully recognized on the face selection screen 740 in Figure 12 are selected as the analysis target, or if a face that failed to be recognized by the trimming frame on the trimming screen in Figure 13 is designated as the analysis target, the confirmation screen 760 shown in Figure 14 is displayed on the display of the mobile terminal 40. The confirmation screen 760 is provided with a face image display area 761, an OK button 762, and a diagnostic image re-selection button 763.

[0136] The face image display area 761 is the area where the face (face image) selected or specified as the analysis target for face analysis is displayed on the face selection screen 740 (Figure 12) or the cropping screen (Figure 13). The user can confirm the face image to be analyzed by checking the face image displayed in the face image display area 761 and pressing the OK button 762.

[0137] The diagnostic image re-selection button 763 is used to re-select a diagnostic image. When the diagnostic image re-selection button 763 is pressed, the image selection screen 730 (Figure 11) is displayed again on the display of the mobile terminal 40.

[0138] When the OK button 762 is pressed, the mobile terminal 40's display shows the analysis screen 770 shown in Figure 15. The analysis screen 770 is a screen that informs the user that the recommendation server 30 is performing (starting) face analysis on the face image that has been determined to be the target of analysis. A "Next" button 771 is provided at the bottom of the analysis screen 770.

[0139] Furthermore, in the center of the analysis screen 770, text and an image are displayed indicating that a facial diagnosis will be performed based on the results of the facial analysis and the answers to the aforementioned question screen. In other words, when the "Proceed" button 771 is pressed, the question screen is displayed on the mobile terminal 40's display.

[0140] <4. Flow of the process for setting the analysis target> Referring to the flowchart in Figure 16, the flow of the analysis target setting process, which is executed on the recommendation server 30 to set the analysis target for face analysis, will be explained. The process in Figure 16 starts when a diagnostic image is selected on the image selection screen 730 (Figure 11) displayed on the mobile terminal 40.

[0141] In step S111, the face recognition unit 532 performs a first face recognition process on the captured image (the diagnostic image selected on the image selection screen 730). The first face recognition process is, for example, a face recognition process using machine learning.

[0142] On the recommendation server 30, the diagnostic images targeted for face recognition and analysis are not the processed images selected on the mobile terminal 40, but rather the unprocessed images that have not undergone the corresponding image processing. Face images extracted from processed images have been deformed by retouching, so feature point detection and other functions are not performed correctly, and the analysis that is intended to be performed cannot be carried out. On the other hand, face images extracted from unprocessed images have not been deformed by retouching, so feature point detection and other functions are performed correctly, making it possible to perform the analysis that is intended to be performed.

[0143] In step S112, the web page provider 531 displays a face selection screen 740 (Figure 12) on the mobile terminal 40, which includes a captured image (diagnostic image) in which a face frame is superimposed on the face that was successfully recognized.

[0144] In step S113, the face recognition unit 532 determines whether or not there are any faces in the captured image (diagnostic image) that were not recognized. If it is determined that there are no faces that were not recognized, the process proceeds to step S114.

[0145] In step S114, the web page provider 531 determines whether or not one of the faces that was successfully recognized has been selected from the captured image (diagnostic image) displayed on the mobile terminal 40. Step S114 is repeated until one of the faces that was successfully recognized is selected, and once one of the faces that was successfully recognized is selected, the process proceeds to step S115.

[0146] In step S115, the face analysis unit 533 analyzes the face selected in the pre-processed image corresponding to the captured image (diagnostic image) displayed on the mobile terminal 40.

[0147] On the other hand, if it is determined in step S113 that there is a face that failed to be recognized, the process proceeds to step S116.

[0148] In step S116, the web page provider 531 determines whether the trimming screen display button 743 has been selected on the face selection screen 740 (Figure 12) displayed on the mobile terminal 40. If it is determined that the trimming screen display button 743 has been selected, the web page provider 531 displays the trimming screen 750 (Figure 13) on the mobile terminal 40 and proceeds to step S117. If the trimming screen display button 743 is not selected, it is assumed that there are faces that have been successfully recognized in the diagnostic image, and the process proceeds to step S114, where it is determined on the face selection screen 740 whether or not a face that has been successfully recognized has been selected.

[0149] In step S117, the web page provider 531 determines whether or not an area has been specified by the trimming frame TF in the captured image (diagnostic image) displayed on the trimming screen 750 on the mobile terminal 40 (whether or not the OK button 753 has been pressed). Step S117 is repeated until an area is specified by the trimming frame TF, and once an area is specified by the trimming frame TF, the process proceeds to step S118.

[0150] In step S118, the face analysis unit 533 performs a second face recognition process on the specified area specified by the trimming frame TF.

[0151] In the second face recognition process, feature points of faces included in the specified region are extracted, and for example, the face with the highest similarity to the face in question is selected from other captured images (pre-processed images) generated in the same game as the diagnostic image. In this way, in the second face recognition process, the face image of the face selected from the other captured images can be the target of face analysis.

[0152] In step S119, the face analysis unit 533 determines whether or not there is only one face in the designated area specified by the trimming frame TF. If there are multiple faces in the designated area, or if there are no faces in the designated area, none of the faces are analyzed, and the process returns to step S117, and the subsequent processing is repeated.

[0153] On the other hand, if it is determined in step S119 that there is only one face included in the designated area, the process proceeds to step S120.

[0154] In step S120, the face analysis unit 533 analyzes faces that are included in the designated area specified by the trimming frame TF. In other words, the face images of faces selected from other captured images in the second face recognition process are the targets of analysis.

[0155] Through the above process, even faces that failed to be recognized can be designated as analysis targets by specifying them within the cropping frame. In other words, even faces that failed to be recognized in previously captured images can be included as targets for face analysis, making face analysis more reliable.

[0156] <5. Variation> The following describes variations of the embodiments described above.

[0157] In the analysis target setting process described above, if the specified area contains multiple faces, none of the faces are included in the analysis. However, it is also possible to present the user with multiple faces (face images) included in the specified area, and have the user select the face image to be included in the analysis.

[0158] In the photo creation game machine 10, eye regions and lip regions are extracted from the captured image in order to perform image processing on the captured image. These eye and lip regions may also be used for face recognition processing performed on the recommendation server 30.

[0159] The above describes an example in which face recognition processing and face analysis are performed by the recommendation server 30 on images captured by the photo creation game machine 10. However, this is not limited to this example; for example, as shown in Figure 17, face recognition processing and face analysis may be performed by a dedicated application (dedicated app) installed on a mobile device on images captured by a user taking a picture of themselves with a mobile device.

[0160] The series of processes described above can be executed by hardware or by software. When the series of processes described above are executed by software, the programs that make up the software are installed from a network or storage medium.

[0161] This recording medium, as shown in Figure 3, for example, consists of a removable media 119, which is distributed separately from the main device body to the administrator of the recommendation server 30 (server 200) for the purpose of distributing the program. This media includes a magnetic disk (including a flexible disk), an optical disk (including CD-ROMs and DVDs), a magneto-optical disk, or semiconductor memory on which the program is recorded. In addition to these, the recording medium also consists of a ROM 212, which is pre-installed in the main device body and contains the program distributed to the administrator, and a hard disk included in the storage unit 218.

[0162] In this specification, the step of describing a program to be recorded on a recording medium includes not only processes that are performed chronologically in the order described, but also processes that are not necessarily performed chronologically, but are executed in parallel or individually.

[0163] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the gist of this technology.

[0164] Furthermore, the effects described herein are merely illustrative and not limiting; other effects may also occur. [Explanation of Symbols]

[0165] 1 Recommendation System, 10 Photo Creation Game Machine, 20 Image Management Server, 30 Recommendation Server, 40 Mobile Terminal, 131 Camera, 132 Touch Panel Monitor, 133 Lighting Device, 151 Printer, 510 Control Unit, 511 Shooting Processing Unit, 512 Image Processing Unit, 513 Printing Processing Unit, 530 Control Unit, 531 Web Page Provisioning Unit, 532 Face Recognition Unit, 533 Face Analysis Unit, 534 Recommendation Information Generation Unit, 535 Data Management Unit, 550 User Database

Claims

1. A recognition unit that recognizes faces in captured images containing people, The analysis department performs facial analysis using faces that have been successfully recognized as the target of analysis. Equipped with, If there are faces that the analysis unit failed to recognize, it will analyze faces selected from other captured images based on the feature points of the faces included in the designated area specified by the user in the captured image. Information processing device.

2. If the analysis unit fails to recognize any of the faces in the captured image containing multiple people, and if multiple faces are included in the designated area, it will not analyze any of the faces. The information processing apparatus according to claim 1.

3. If the analysis unit fails to recognize any of the faces and none of the faces are included in the designated area, it will not analyze any of the faces. The information processing apparatus according to claim 1 or 2.

4. The recognition unit recognizes the face by performing a first face recognition process on the captured image, If there is a face that failed to be recognized and the face is included in the designated area, the analysis unit will analyze the face recognized by a second face recognition process, which is different from the first face recognition process, for the designated area. The information processing apparatus according to claim 1.

5. If the analysis unit successfully recognizes multiple faces in the captured image in which multiple people are present, it will analyze the face selected by the user in the captured image. The information processing apparatus according to claim 1.

6. The aforementioned captured image is the image selected by the user from among multiple images generated by the photo creation game machine. The information processing apparatus according to any one of claims 1 to 5.

7. The designated area is the area specified by the user in the captured image displayed on the user's mobile device. The information processing apparatus according to claim 1.

8. Information processing device, Recognizing faces in photographs containing people, We perform facial analysis on faces that have been successfully recognized. If there are faces that fail to be recognized, the face selected from other captured images will be analyzed based on the feature points of the face included in the designated area specified by the user in the captured image. Information processing methods.

9. On the computer, Recognizing faces in photographs containing people, We perform facial analysis on faces that have been successfully recognized. If there are faces that fail to be recognized, the face selected from other captured images will be analyzed based on the feature points of the face included in the designated area specified by the user in the captured image. A program to execute a process.

Images