Information processing device, information processing method, and program

Abstract

This technology enables the estimation of a person to be evaluated from a scene in a captured image, without identifying a specific individual, and evaluates their facial expression. [Solution] The information processing device includes: detection means for obtaining positional information of a person's face region detected from an image and obtaining the area of ​​the face region from the positional information; determination means for determining the scene in which the image was taken based on the area of ​​the face region and the positional relationship of the face region; determination means for determining the main subject in the image based on the scene determined by the determination means; and evaluation means for evaluating the facial expression of the main subject determined by the determination means.

Description

Technical Field

[0001] The present invention relates to a technique for estimating and evaluating a main subject from a scene of a photographed image.

Background Art

[0002] A function for evaluating the expression and quality of an image by image analysis using AI technology is being realized. Elements for evaluating an image include expressions such as smiling and winking based on the face of a person in the image, and defocus or camera shake and subject blur. Also, when evaluating an image, for example, when a single image contains the faces of multiple people and the expressions of each person are different, it is necessary to determine which person's face to use as a reference for evaluating the image.

[0003] Patent Document 1 describes a photo classification method for classifying whether a given photo is a single photo of one reference person or a group photo of multiple people including the reference person, and determining whether it includes unspecified people other than the reference person in the case of a group photo. Also, Patent Document 2 describes a method for detecting a person commonly included in a plurality of images and estimating a main subject based on the position or line-of-sight direction of the person in the plurality of images.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Patent Document 1 requires the registration of a person's facial image in advance to identify a reference person. Patent Document 2 requires multiple images to estimate the main subject and to identify individuals included in those multiple images. However, facial feature information of individuals may be considered sensitive personal information, and in some countries and regions, there are strict restrictions on the acquisition and management of such information, making image evaluation by individual identification difficult. Furthermore, even if an image features a person, that person may not be someone the photographer knows, making it difficult to perform image evaluation by identifying the person to be evaluated in advance in such cases.

[0006] This invention has been made in view of the above problems, and its objective is to realize a technology that estimates the person to be evaluated from a scene in a captured image without identifying a specific individual, and evaluates the facial expression. [Means for solving the problem]

[0007] To solve the above problems and achieve the objective, the information processing device of the present invention includes: detection means for obtaining positional information of a person's face region detected from an image and obtaining the area of ​​the face region from the positional information; determination means for determining the scene in which the image was taken based on the area of ​​the face region and the positional relationship of the face region; determination means for determining the main subject in the image based on the scene determined by the determination means; and evaluation means for evaluating the facial expression of the main subject determined by the determination means. [Effects of the Invention]

[0008] According to the present invention, it is possible to estimate the person to be evaluated from the scene of a captured image and evaluate their facial expression without identifying a specific individual. [Brief explanation of the drawing]

[0009] [Figure 1] System configuration diagram of this embodiment. [Figure 2]A block diagram illustrating the hardware configuration of the information processing device and server device of this embodiment. [Figure 3] A block diagram illustrating the hardware configuration of the imaging device according to this embodiment. [Figure 4] A block diagram illustrating the functional configuration of the facial expression evaluation device in the server device of this embodiment. [Figure 5] A diagram illustrating the image to be evaluated in this embodiment. [Figure 6] A diagram illustrating the configuration of the face region management table in this embodiment. [Figure 7] A diagram illustrating the focus position and line of sight direction in the image to be evaluated in this embodiment. [Figure 8] This figure illustrates a method for calculating the distance variation deviation of the face region in the image to be evaluated in this embodiment. [Figure 9] This figure illustrates a method for calculating the deviation of the alignment of face regions in the image to be evaluated in this embodiment. [Figure 10] A diagram illustrating the configuration of the scene determination criteria table in this embodiment. [Figure 11] A diagram illustrating the scene determination method of this embodiment. [Figure 12] A diagram illustrating the scene determination method of this embodiment. [Figure 13] A diagram illustrating the scene determination method of this embodiment. [Figure 14] A diagram illustrating the configuration of the evaluation target determination table in this embodiment. [Figure 15] A flowchart illustrating the image evaluation process of this embodiment. [Figure 16] A flowchart illustrating the face position detection process of this embodiment. [Figure 17] A flowchart illustrating the facial region information analysis process of this embodiment. [Figure 18] A flowchart illustrating the scene determination process of this embodiment. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant explanations are omitted.

[0011] In the present embodiment, an example will be described in which an image captured by an imaging device is uploaded to a server device and evaluation processing of the image is performed in the server device in a system in which the imaging device, the information processing device, and the server device are communicably connected via a network.

[0012] <System Configuration> First, the system configuration of the present embodiment will be described with reference to FIG. 1.

[0013] FIG. 1 is a diagram illustrating the system configuration of the present embodiment.

[0014] The user in the present embodiment is a person who uses the system of the present embodiment.

[0015] The system of the present embodiment includes an imaging device 101, an information processing device 102, and a server device 103. The imaging device 101, the information processing device 102, and the server device 103 are communicably connected via a network 104.

[0016] The imaging device 101 captures image data to be processed in the present embodiment and uploads it from the user's local environment to the server device 103 via the network 104. The imaging device 101 in the present embodiment is a digital camera, but can be widely applied to electronic devices having a photographing function. Such electronic devices include computer devices (including personal computers, tablet computers, smartphones, etc.). The image data to be processed in the present embodiment is not limited to a specific format such as a file format or an encoding format.

[0017] The information processing device 102 is a terminal device used by a user of the system in this embodiment. The information processing device 102 transmits processing instructions to the server device 103 via the network 104, receives information transmitted from the server device 103, and displays the information received from the server device 103. The information processing device 102 in this embodiment is a general-purpose computer such as a personal computer (PC) or tablet computer, or a smartphone.

[0018] The server device 103 performs evaluation processing on image data uploaded from the imaging device 101 via the network 104 and transmits the evaluation results to the information processing device 102. In this embodiment, the server device 103 is an information processing device such as a cloud server that provides services through cloud computing.

[0019] The communication method of network 104 may be either wireless communication or wired communication.

[0020] The system of this embodiment is not limited to a configuration in which image data is transmitted to the server device 103, but may also be transmitted in a form other than communication over a network. For example, image data may be uploaded directly from the imaging device 101 to the server device 103, or it may be uploaded from the imaging device 101 to the server device 103 via the information processing device 102.

[0021] The server device 103 is not limited to the form of a server; it may also be an electronic device having similar functions to the information processing functions of a server.

[0022] Furthermore, in this embodiment, the output destination for the image evaluation results from the server device 103 is the information processing device 102, but this configuration is not limited to this depending on the format of the output data. For example, if a server of another system is connected, output of data in text file format or displayed in a UI is not necessary, and the data may be output in a format such as a database.

[0023] Furthermore, although the image evaluation process is performed by the server device 103 in this embodiment, the system is not limited to this configuration, and the image evaluation process may be performed by the information processing device 102.

[0024] <Device configuration> Next, with reference to Figures 2 to 4, the configurations of the imaging device 101, information processing device 102, and server device 103 that constitute the system of this embodiment will be described.

[0025] Figure 2 is a block diagram illustrating the configuration of the information processing device 102 and server device 103 in this embodiment.

[0026] The information processing device 102 and server device 103 of this embodiment include a display 201, VRAM 202, BMU 203, keyboard 204, pointing device 205, CPU 206, ROM 207, RAM 208, HDD 209, removable storage device 210, network I / F 211, and bus 212.

[0027] The display 201 shows images, icons, messages, menus, and other user interface information, for example, for managing the information processing device 102 and the server device 103.

[0028] VRAM202 is video memory where data for display on display 201 is drawn. The data generated in VRAM202 is transferred to display 201 according to predetermined rules, thereby displaying an image on display 201.

[0029] The BMU (Bit Move Unit) 203 is a circuit that controls data transfer between memories (e.g., between VRAM 202 and other memories) and between memory and each I / O device (e.g., network I / F 211).

[0030] The keyboard 204 is an operating unit that has various keys for inputting characters and other information.

[0031] The pointing device 205 is used, for example, to point to icons, menus, or other content displayed on the display 201, or for dragging and dropping objects.

[0032] The CPU206 is a control unit that controls each component based on the OS (operating system) and control programs (applications) stored in the ROM207 and HDD209.

[0033] ROM207 is a non-volatile memory that stores control programs and data executed by CPU206.

[0034] RAM208 is a work memory that contains the work area for CPU206, a data storage area for error handling, and a control program load area.

[0035] HDD209 is an auxiliary storage device that stores control programs, content, and data executed by CPU206.

[0036] The removable storage device 210 is a high-capacity storage device that stores the OS and control programs.

[0037] The network interface 211 is a communication unit that communicates with external devices such as the imaging device 101 and the information processing device 102. Furthermore, if the external device is a communication device such as a smartphone, the network interface 211 can perform public wireless communication such as 4G or 5G.

[0038] Bus 212 includes an address bus, a data bus, and a control bus.

[0039] The control program executed by the CPU 206 can be provided from the ROM 207, HDD 209, removable storage device 210, or from an external device connected via the network interface 211.

[0040] The control program includes the operating system (OS), which is the basic software executed by the CPU 206, and applications that work in cooperation with the OS to realize advanced functions. The applications include those that allow the information processing device 102 and the server device 103 of this embodiment to execute the flowchart processing described later.

[0041] The flowchart processing in this embodiment is realized by loading software provided by an application. The application is assumed to have software for utilizing the basic functions of the operating system installed on the information processing device 102 and the server device 103. The operating systems of the information processing device 102 and the server device 103 may also have software for realizing the processing in this embodiment.

[0042] Figure 3 is a block diagram illustrating the configuration of the imaging device 101 in this embodiment.

[0043] The photographic lens 301 is a lens group that includes a zoom lens and a focus lens. The shutter 302 has an aperture function. The imaging unit 303 is an image sensor composed of a CCD or CMOS, etc., which converts the optical image of the subject into an electrical signal. The A / D converter 304 converts an analog signal into a digital signal. The A / D converter 304 is used to convert the analog signal output from the imaging unit 303 into a digital signal. The barrier 305 covers the imaging system of the imaging device 101, including the photographic lens 301, thereby preventing dirt and damage to the imaging system, including the photographic lens 301, shutter 302, and imaging unit 303.

[0044] The image processing unit 306 performs resizing and color conversion processing, such as predetermined pixel interpolation and reduction, on the data from the A / D converter 304 or the data from the memory control unit 307. In addition, the image processing unit 306 performs predetermined calculations using the captured image data, and the system control unit 308 performs exposure control and distance measurement control based on the obtained calculation results.

[0045] The output data from the A / D converter 304 is written to the memory 309 via the image processing unit 306 and the memory control unit 307, or via the memory control unit 307. The memory 309 stores image data obtained by the imaging unit 303 and converted into digital data by the A / D converter 304, as well as image data for display on the display unit 310. The memory 309 has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of video, and audio. The memory 309 also serves as a memory for image display (video memory).

[0046] The D / A converter 311 converts the image display data stored in the memory 309 into an analog signal and supplies it to the display unit 310. In this way, the display image data written to the memory 309 is displayed by the display unit 310 via the D / A converter 311. The display unit 310 displays the analog signal from the D / A converter 311 on a display device such as an LCD panel. The digital signal, which has been A / D converted by the A / D converter 304 and stored in the memory 309, is converted to analog by the D / A converter 311 and sequentially transferred to the display unit 310 for display, thereby enabling the display of a live view image.

[0047] The non-volatile memory 312 is an electrically erasable and restorable memory, such as an EEPROM. Constants for the operation of the system control unit 308, programs, and the like are stored in the non-volatile memory 312.

[0048] The system control unit 308 controls the entire imaging device 101. The system control unit 308 controls each component by executing a program stored in the non-volatile memory 312. RAM is used for the system memory 313, and constants, variables for the operation of the system control unit 308, and programs read from the non-volatile memory 312 are loaded into it. The system control unit 308 also performs display control by controlling the memory 309, the D / A converter 311, the display unit 310, etc.

[0049] The mode selector switch 314, the first shutter switch 315, the second shutter switch 316, and the operation unit 317 are operating means for inputting various operation instructions to the system control unit 308.

[0050] The mode selector switch 314 switches the operating mode of the system control unit 308 to one of the following: still image shooting mode, video recording mode, playback mode, etc.

[0051] The first shutter switch 315 turns on during the operation of the shutter button on the imaging device 101, specifically when it is half-pressed (instruction to prepare for shooting), and generates the first shutter switch signal SW1. The first shutter switch signal SW1 initiates shooting preparation processes such as AF processing, AE processing, AWB processing, and EF processing.

[0052] The second shutter switch 316 turns on when the shutter button operation is completed, so-called full press (shooting instruction), and generates the second shutter switch signal SW2. The system control unit 308 starts a series of shooting processes, from reading the signal from the imaging unit 303 to writing the image data to the recording medium 325, in response to the second shutter switch signal SW2.

[0053] Each operating element of the control unit 317 is assigned a function as appropriate for each scene by selecting various function icons displayed on the display unit 310, and acts as various function buttons. Examples of function buttons include an exit button, back button, image advance button, jump button, filter button, and attribute change button. For example, when the menu button is pressed, various configurable menu screens are displayed on the display unit 310. The user can intuitively make various settings using the menu screen displayed on the display unit 310 and the four directional buttons (up, down, left, and right) and the SET button.

[0054] The controller wheel 318 is a rotatable operating element included in the operating unit 317 and is used in conjunction with the directional buttons to indicate selection items. When the controller wheel 318 is rotated, an electrical pulse signal is generated according to the amount of rotation, and the system control unit 308 controls each component of the imaging device 101 based on this pulse signal. This pulse signal allows the system control unit 308 to determine the angle by which the controller wheel 318 has been rotated and how many rotations it has made. The controller wheel 318 can be any operating element that can detect rotation. For example, it could be a dial operating element in which the controller wheel 318 itself rotates in response to the user's rotation and generates a pulse signal. Alternatively, it could be an operating element consisting of a touch sensor, in which the controller wheel 318 itself does not rotate, but detects the rotational movement of the user's finger on the controller wheel 318.

[0055] The controller ring 319 is a rotatable operating member included in the operating unit 317, and is an operating member that can be rotated around the optical axis of the lens barrel. When the controller ring 319 is operated, an electrical pulse signal corresponding to the amount of rotation (operation amount) is generated, and the system control unit 308 controls each component of the imaging device 101 based on this pulse signal. In addition, when the function switching button of the controller ring 319 is pressed, a menu screen that allows the function assigned to the controller ring 319 to be changed is displayed on the display unit 310.

[0056] The controller wheel 318 and controller ring 319 are used to select setting items and change their values.

[0057] The power switch 320 is an operating component that switches the power of the imaging device 101 on or off.

[0058] The power control unit 321 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are energized, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery level. Furthermore, the power control unit 321 controls the DC-DC converter based on the detection results and instructions from the system control unit 308, and supplies the necessary voltage to each component, including the recording medium 325, for the required period of time.

[0059] The power supply unit 322 consists of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries and Li-ion batteries, and an AC adapter. The recording medium I / F 324 is an interface with the recording medium 325, such as a memory card or hard disk. The recording medium 325 is a recording medium such as a memory card for recording captured images, and is composed of semiconductor memory, magnetic disks, etc.

[0060] The communication unit 323 connects to an external device via a wireless antenna or wired cable to enable communication and transmit / receive image data. The communication unit 323 can also connect to a wireless LAN (Local Area Network) or the internet. The communication unit 323 can transmit image data (including live view images) captured by the imaging unit 303 and image files recorded on the recording medium 325 to an external device, and can also receive image data and other various information from an external device. The communication unit 323 may use wireless communication modules such as infrared communication, Bluetooth®, Bluetooth® Low Energy, WirelessUSB, or wired connection means such as a USB cable, HDMI®, or IEEE1394, in addition to wireless LAN.

[0061] The network interface 326 is controlled by the system control unit 308 and communicates with external devices via the network.

[0062] Note that the configuration of the imaging device 101 in this embodiment is just one example, and among the components in Figure 3, the shutter 302, mode switching switch 314, display unit 310, etc., are not essential.

[0063] Figure 4 is a block diagram illustrating the functional configuration of the facial expression evaluation device 400 in the server device 103 of this embodiment.

[0064] Each function of the facial expression evaluation device 400 in this embodiment is realized by utilizing the hardware resources and control program (application) described in Figure 2.

[0065] The facial expression evaluation device 400 receives image data transmitted via the network 104 with the data receiving unit 401 and stores it in the image data storage 402.

[0066] The image data reading unit 403 reads the instructed image data from multiple image data stored in the image data storage 402 and extracts metadata such as shooting information recorded in that image data.

[0067] The face position detection unit 405 detects whether a person's face is included in the image read by the image data reading unit 403. It obtains the coordinate information of the detected face region, calculates the area ratio, which is the ratio of the area of ​​the face region to the total area of ​​the image, and stores it in the face region information database (DB) 404.

[0068] The face region information analysis unit 406 obtains the coordinate information and area ratio of the face region detected by the face position detection unit 405 from the face region information DB 404, sorts the face region information in descending order of area ratio, and calculates the difference in area ratio between preceding and succeeding face regions.

[0069] The scene determination unit 407 estimates the scene in which the image was taken based on the area and positional relationship of the faces of people included in the image. The scene determination unit 407 determines the scene in which the image was taken by comparing the face area information recorded in the face area information DB 404 with predetermined determination criteria.

[0070] The evaluation target determination unit 408 determines one or more facial regions to be evaluated based on the determination criteria defined for each scene determined by the scene determination unit 407.

[0071] The image evaluation unit 409 performs facial expression evaluations, such as smiles, closed eyes, and blur / blur, on the facial region to be evaluated, as determined by the evaluation target determination unit 408, and outputs an evaluation of the image's overall quality. The image evaluation unit 409 excludes facial expression evaluations if it determines that the person is not the main subject in the image.

[0072] Figure 5 illustrates the image 500 to be evaluated in this embodiment.

[0073] The image 500 captured by the imaging device 101 and transmitted to the server device 103 is recorded in the image data storage 402 as an image in which a person 501, which is the main subject, and a person 502 in the background area are visible.

[0074] The facial expression evaluation device 400 detects the face region 503 of person 501 in the face position detection unit 405, approximates the detected face region 503 as a rectangular region, and obtains the top-left coordinate 504 and bottom-right coordinate 505, which represent the rectangular region, as position information for the face region 503. Similarly, it detects the face region 506 of person 502 in the background region, approximates the detected face region 506 as a rectangular region, and obtains the top-left coordinate 507 and bottom-right coordinate 508, which represent the rectangular region, as position information for the face region 506. Coordinate 509 indicates the image size of image 500.

[0075] Figure 6 illustrates a face region management table 600 containing face region information for each image to be evaluated in this embodiment.

[0076] The facial region management table 600 is stored in the facial region information DB 404 of the facial expression evaluation device 400.

[0077] When the face position detection unit 405 detects a face region from an image, it assigns a unique face region ID 601 to that region, and adds the upper-left coordinates 602 and lower-right coordinates 603 of a rectangular region that approximates the detected face region to the face region management table 600 as face region information. Since multiple face regions are allowed to exist in a single image, face region information is added to the face region management table 600 for each detected face region.

[0078] Next, the face region information analysis unit 406 calculates the ratio (area ratio) of the number of pixels in each face region obtained from the coordinate information 602 and 603 for each face region ID 601 to the total number of pixels in the entire image, and records it in the region area ratio 604. The total number of pixels in the entire image corresponds to the coordinate information 509 shown in Figure 5, for example.

[0079] Furthermore, the face region information analysis unit 406 uses the center or focus position of the image as a reference position to calculate the distance from the reference position in the image to each face region, and records it as face region information at the reference distance 605.

[0080] The priority coefficient 606 is a coefficient used by the image data reading unit 403 to assign priority to each face region based on the image metadata acquired. The metadata includes, for example, the focus position (AF point) for AF processing and the line of sight vector of the person's face region.

[0081] After reading all the face region information contained in the image, the face region information analysis unit 406 sorts this face region information in descending order of area ratio 604. It calculates the difference in area ratio 604 between the face region information before and after sorting, and sets the area ratio threshold flag 607 to true for face regions where the difference changes rapidly, and records this information. In this embodiment, face region information for which the area ratio threshold flag 607 is set to true becomes a threshold for determining whether the person can be considered the main subject in the image or a person that is in the background.

[0082] Figure 7 illustrates an image 700 of a person 701 that includes focus position information and gaze information detected by the image data reading unit 403 of the facial expression evaluation device 400 of this embodiment.

[0083] The imaging device 101 records information on all focus positions 702 and information on the focus position 703 that was achieved by AF as metadata for the image 700.

[0084] The image data reading unit 403 of the facial expression evaluation device 400 acquires metadata of the image 700 or coordinate information of the focus position 703, and the face position detection unit 405 determines whether the focus position 703 is within the range specified by the face region coordinate information 602, 603. If the focus position 703 is included in the face region coordinate information 602, 603, it can be determined that the photographer considers that face region to be the main subject, and the priority coefficient 606 of that face region ID 601 is recorded. Alternatively, the face position detection unit 405 analyzes the gaze vector of the face region of person 701 in the image 700 and records a priority coefficient 606 based on the gaze direction 704 of the face region of person 701.

[0085] Figure 8 illustrates a method for calculating the distance variation deviation of the face region in the image 800 to be evaluated in this embodiment.

[0086] The face position detection unit 405 detects the face regions 804 and 805 of people 802 and 803 from the image 800 and records the coordinate information 602 and 603 in the face region management table 600.

[0087] The face region information analysis unit 406 determines the center coordinates of image 800, calculates the distance between the center coordinates of image 800 and the center coordinates of each face region 804, 805 recorded in the face region management table 600, and records this as the deviation of the distance variation of the face regions in the reference distance 605 of the face region management table 600. The reference position can be the coordinate information of the center of the image, or the coordinate information of the focus position 703 shown in Figure 7.

[0088] Figure 9 illustrates a method for calculating the deviation of the alignment of the face region in the image 900 to be evaluated in this embodiment.

[0089] The face position detection unit 405 detects the face regions 904 and 905 of people 902 and 903 from the image 900 and records the coordinate information 602 and 603 in the face region management table 600.

[0090] The face region information analysis unit 406 obtains the center coordinates of the face regions 904 and 905 detected by the face position detection unit 405, and calculates the difference 906 in the Y direction of these center coordinates as the deviation of alignment variation in the direction in which the face regions are aligned. Note that instead of calculating the difference in the Y direction of the face regions, the difference in the X direction may be calculated, or the difference in both the X and Y directions may be calculated.

[0091] Figure 10 illustrates a scene determination criteria table 1000, which records the criteria for determining the scene in which the image to be evaluated in this embodiment was taken.

[0092] The scene determination unit 407 of the facial expression evaluation device 400 determines the scene in which the image to be evaluated was taken by comparing the information obtained from the facial region information recorded in the facial region management table 600 with each item of the scene determination criteria 1000.

[0093] Scene type 1001 defines a shooting scene to be estimated in advance. In this embodiment, examples of shooting scenes include "portrait," "small group scene," "large group scene," and "crowd scene."

[0094] The scene determination unit 407 reads the face region management table 600 of the image to be evaluated from the face region information DB 404 and compares the area ratio 604 with the maximum face region area ratio 1002 in the scene determination criteria table 1000. The scene determination unit 407 also calculates the number of face regions where the area ratio 604 exceeds the threshold, using the face region information for which the area ratio threshold flag 607 is set to true in the face region management table 600 as a threshold, and compares this with the number of face regions 1003 that exceeds the area ratio threshold in the scene determination criteria table 1000. In addition, the scene determination unit 407 compares the deviation of the variation in face region distance calculated from the reference distance 605 in the face region management table 600 with the deviation of the reference distance 1004 in the scene determination criteria table 1000. Furthermore, the scene determination unit 407 compares the deviation of the alignment of the face region calculated from the difference 906 in the Y direction of the face region in Figure 9 with the deviation of the alignment of the standard in the scene determination criterion table 1000.

[0095] Based on these comparison results, the scene determination unit 407 determines which of the predefined scene types 1001 the scene in which the image to be evaluated was taken corresponds to.

[0096] Each judgment criterion defined in the scene judgment criterion table 1000 can be set using a combination of numerical values, default values, and thresholds.

[0097] Figure 11 illustrates an image 1100 that is determined to be a scene with a small number of people by the scene determination unit 407 of the facial expression evaluation device 400 of this embodiment.

[0098] The face position detection unit 405 detects the face regions 1105, 1106, 1107, and 1108 of four people, 1101, 1102, 1103, and 1104, as subjects in the image, and records the detected face region information in the face region management table 600. The face region information analysis unit 406 sorts the face region information recorded in the face region management table 600 in descending order of area ratio, rearranges it according to priority, and then compares the value obtained from each face region information in the face region management table 600 with the judgment criteria in the scene judgment criteria table 1000. If the result satisfies the following conditions, it is determined to be a scene with a small number of people. • The maximum value of area ratio 604 must be sufficiently large in the largest face area ratio 1002 of the scene determination criteria table 1000 (the second largest value in the example in Figure 10). • The number of face regions where the area ratio 604 exceeds the area ratio threshold is sufficiently small compared to the number of face regions exceeding the area ratio threshold (1003) in the scene determination criteria table 1000 (the second smallest value in the example in Figure 10). • The distance variation deviation in the face region must be sufficiently small compared to the reference distance variation deviation 1004 in the scene determination criterion table 1000 (the second smallest value in the example in Figure 10). • The deviation in the alignment of the face region must be sufficiently small compared to the deviation in alignment 1005 of the scene judgment criteria table 1000 (the second smallest value in the example in Figure 10). Figure 12 illustrates an image 1200 that is determined to be a scene with multiple people by the scene determination unit 407 of the facial expression evaluation device 400 of this embodiment.

[0099] The face position detection unit 405 detects multiple face regions 1202, including person 1201, as subjects in the image, and records the detected face region information in the face region management table 600. The face region information analysis unit 406 sorts the face region information recorded in the face region management table 600 in descending order of area ratio, rearranges them according to priority, and then compares the values ​​obtained from each face region information in the face region management table 600 with the judgment criteria in the scene judgment criteria table 1000. If the result satisfies the following conditions, it determines that it is a scene with many people. • The maximum value of area ratio 604 must be sufficiently small in the largest face area ratio 1002 of the scene determination criterion table 1000 (the third largest value in the example in Figure 10). • The number of face regions where the area ratio 604 exceeds the area ratio threshold is sufficiently large in the scene determination criteria table 1000, which is 1003 (the third largest value in the example in Figure 10). • The deviation in the distance variation of the face region is sufficiently large compared to the deviation in the distance variation of the reference in the scene judgment criterion table 1000 (the third largest value in the example in Figure 10). • The deviation of the alignment variability of the face region is sufficiently large compared to the deviation of the alignment variability 1005 of the scene judgment criterion table 1000 (the third largest value in the example in Figure 10). Figure 13 illustrates an image 1300 that is determined by the scene determination unit 407 of the facial expression evaluation device 400 of this embodiment to be a crowd scene including a person or a landscape scene.

[0100] The face position detection unit 405 detects the face regions 1308, 1309, 1310, 1311, 1312, 1313, and 1314 of the people 1301 and 1304 located relatively close to the viewer, the people 1305, 1302, and 1303 located behind them, and the people 1310 and 1311 located further away, and records the detected face region information in the face region management table 600. The face region information analysis unit 406 sorts the face region information recorded in the face region management table 600 in descending order of area ratio, rearranges it according to priority, and then compares the value obtained from each face region information in the face region management table 600 with the judgment criteria in the scene judgment criteria table 1000. If the result satisfies the following conditions, it determines that there are people in the image, but they are not considered the main subjects in a crowd scene or landscape scene. • The maximum value of area ratio 604 is very small in the largest face area ratio 1002 of the scene determination criterion table 1000 (the smallest value in the example in Figure 10). • The number of face regions exceeding the area ratio threshold (area ratio 604) is very large in the scene determination criteria table 1000 (number of face regions exceeding the area ratio threshold 1003) (the largest value in the example in Figure 10). • The deviation in distance variation of the face region is very large at the reference distance variation deviation 1004 of the scene judgment criterion table 1000 (the largest value in the example in Figure 10). • The deviation in the alignment of the face region is very large at the deviation in the alignment of the standard in scene judgment criterion table 1000 (the largest value in the example in Figure 10). In this embodiment, the items used to determine a scene include the area ratio of the largest face region, the number of face regions whose area ratio exceeds a threshold, the deviation in the variation of the distance between face regions, and the deviation in the variation of the alignment of face regions. However, a scene may be determined based on at least one of these items.

[0101] Figure 14 illustrates an evaluation target determination table 1400 used by the evaluation target determination unit 408 of the facial expression evaluation device 400 in this embodiment to determine which facial region in an image will be evaluated based on the shooting scene determined by the scene determination unit 407.

[0102] In this embodiment, an example of evaluating facial expressions as "smile," "closed eyes," and "blur / shake" is described. Based on the shooting scene determination determined by the scene determination unit 407, the criteria for the facial region to be evaluated when determining smile 1402, closed eyes 1403, and blur / shake 1404 are defined for each predefined shooting scene 1401.

[0103] In this embodiment, smile detection is performed in portraits and small group scenes, but not in large group scenes, crowded scenes, or landscape scenes. Furthermore, closed eyes detection is performed in portraits, small group scenes, and large group scenes, but not in crowded scenes or landscape scenes. In addition, blur / blur detection is performed in portraits, small group scenes, and large group scenes, but not in crowded scenes or landscape scenes.

[0104] In portrait scenes, one or more facial regions whose area ratio exceeds a threshold are targeted for smile detection. In scenes with a small number of people, a predetermined percentage or more of the facial regions whose area ratio exceeds the threshold are targeted for smile detection.

[0105] Furthermore, in portrait scenes, all facial regions exceeding the threshold in area ratio are targeted for eye-closing detection; in scenes with a small number of people, one or more facial regions exceeding the threshold in area ratio are targeted; and in scenes with a large number of people, all facial regions are targeted for eye-closing detection.

[0106] Furthermore, in portrait scenes, all face regions exceeding the threshold in area ratio are targeted for blur / blur detection; in small group scenes, one or more face regions exceeding the threshold in area ratio are targeted; and in large group scenes, all face regions are targeted for blur / blur detection.

[0107] For example, in a portrait with one or two main subjects, smile detection requires that at least one of the main subjects is smiling, eye-closed detection requires that everyone has their eyes open, and blur / blur detection requires that everyone is not blurred or blurred.

[0108] In scenes with a small group of 3 to 5 people, the following conditions must be met: for smile detection, a separately specified percentage of the facial area larger than a certain predetermined value must be smiling; for eye-closed detection, at least one person must not have their eyes closed; and for blur / out-of-focus detection, at least one person must not be blurry or out of focus.

[0109] In scenes with more than 10 people, smiles are not expected to be required, so smile detection will be excluded from evaluation. On the other hand, eye-closed detection requires that the eyes be open in all facial regions recognized as face areas, and similarly, blur / blur detection requires that all facial regions are not blurred or blurred.

[0110] In crowded or landscape scenes, people are not considered the primary subject of the image, and facial expression detection is excluded from evaluation in all evaluation criteria.

[0111] Figure 15 is a flowchart illustrating the image evaluation process of the facial expression evaluation device 400 in the server device 103 of this embodiment.

[0112] In Figure 15, the CPU 206 of the server device 103 executes a program to control each component and realize the functions of the facial expression evaluation device 400.

[0113] In step S1501, the CPU 206 of the server device 103, acting as the data receiving unit 401 of the facial expression evaluation device 400, receives image data uploaded from the imaging device 101 or the information processing device 102 into the memory 208 or storage 209.

[0114] In step S1502, the CPU 206 of the server device 103 saves the image data received in step S1501 to the image data storage 402 of the facial expression evaluation device 400.

[0115] In step S1503, the CPU 206 of the server device 103 reads the image data saved in step S1502, retrieves metadata attached to the image data, and records it in the face area information DB 404.

[0116] In step S1504, the CPU 206 of the server device 103, acting as the face position detection unit 405 of the facial expression evaluation device 400, performs face position detection on the image data read in step S1503. The face position detection unit 405 of the facial expression evaluation device 400 acquires the coordinate information 504, 505 of the face region, the coordinate information of the focus position 703, and the gaze vector 704, and records them in the face region management table 600 of the face region information DB 404. The face position detection unit 405 also updates the face region management table 600 by setting a priority coefficient 606 from the coordinate information of the focus position 703 and the gaze vector 704.

[0117] Furthermore, if no region is found to be a person's face during face position detection, the image data received in step S1501 is determined not to contain a person, and processing is terminated.

[0118] In step S1505, the CPU 206 of the server device 103, acting as the face region information analysis unit 406 of the facial expression evaluation device 400, reads the coordinate information 602 and 603 of the face region from the face region management table 600 updated in step S1505, calculates the area ratio 604, the reference distance 605, and the difference 906 in the Y direction of the face region, sets the area ratio threshold 607, and records it in the face region management table 600.

[0119] In step S1506, the CPU 206 of the server device 103, acting as the scene determination unit 407 of the facial expression evaluation device 400, obtains the number of face regions exceeding the area ratio threshold, the deviation of the variation in the distance between face regions, and the deviation of the variation in the alignment of face regions from the face region management table 600 recorded in step S1505, and determines the scene in which the image was taken by comparing it with the corresponding determination criteria 1002 to 1005 in the scene determination criterion table 1000.

[0120] In step S1507, the CPU 206 of the server device 103, acting as the evaluation target determination unit 408 of the facial expression evaluation device 400, reads facial region information from the facial region management table 600 recorded in step S1504, and determines the facial region to be evaluated by comparing it with the evaluation target determination table 1400 based on the shooting scene determined in step S1506, and then performs the facial expression evaluation.

[0121] In step S1508, the CPU 206 of the server device 103, acting as the image evaluation unit 409, transmits the image evaluation results performed in step S1507 to the information processing device 102. The image evaluation results may be used as parameters in other processing in the server device 103.

[0122] Figure 16 illustrates the face position detection process in step S1504 of Figure 15.

[0123] In step S1601, the face position detection unit 405 detects a person's face in the image data read from the image data storage 402 of the facial expression evaluation device 400 in step S1503.

[0124] In step S1602, the face position detection unit 405 generates a face region ID 601 for the face region detected in step S1601.

[0125] In step S1603, the face position detection unit 405 acquires coordinate information for the upper left corner coordinate 504 and the lower right corner coordinate 505, which determine the rectangular area of ​​the face region detected in step S1601.

[0126] In step S1604, the face position detection unit 405 adds the face region ID generated in step S1602 and the face region coordinate information acquired in step S1603 to the face region management table 600.

[0127] In step S1605, the face position detection unit 405 determines whether the face region detected in step S1601 matches the focus position 703. The face position detection unit 405 determines whether the coordinate information of the focus position 703 from the metadata of the image data acquired in step S1503 is included in the rectangular area of ​​the face region acquired in step S1603. If it is included, the process proceeds to step S1606; otherwise, the process proceeds to step S1607.

[0128] In step S1606, the face position detection unit 405 sets the priority coefficient 606 of the face region management table 600. In this embodiment, the priority coefficient is set so that face regions that match the focus position 703 of the image or face regions whose gaze direction 704 is facing forward have a higher priority.

[0129] In step S1607, the face position detection unit 405 determines whether or not there are unprocessed face regions in the image data. If there are no unprocessed face regions, it terminates the process. If there are unprocessed face regions, it returns to step S1602 and records all face regions of multiple people included in the image in the face region management table 600.

[0130] Figure 17 illustrates the facial region information analysis process in step S1505 of Figure 15.

[0131] In step S1701, the face region information analysis unit 406 sorts the face region information recorded in the face region management table 600 in step S1604 of Figure 16 in descending order of area ratio 604.

[0132] In step S1702, the face region information analysis unit 406 performs a process to rearrange the face region information in the face region management table 600, which was sorted in step S1701, in descending order of priority coefficient 606.

[0133] In step S1703, the face region information analysis unit 406 calculates the difference in area ratio 604 before and after sorting for the information of each face region in the face region management table 600, which was sorted in step S1701 based on the area ratio 604.

[0134] In step S1704, the face region information analysis unit 406 calculates the amount of change in the difference of the area ratio obtained in step S1703, and sets the area ratio threshold flag 607 to true for face regions where the difference changes rapidly, and records it.

[0135] In step S1705, the face region information analysis unit 406 calculates the reference distance 605 and records it in the face region management table 600.

[0136] In step S1706, the face region information analysis unit 406 calculates the deviation of the distance variation of the face region based on the reference distance 605 obtained in step S1705.

[0137] In step S1708, the face region information analysis unit 406 calculates the deviation of the alignment of the face region from the coordinates 602 and 603 of the face region acquired in step S1603.

[0138] Figure 18 illustrates the scene determination process in step S1506 of Figure 15.

[0139] In step S1801, the scene determination unit 407 obtains the maximum value of the area ratio 604 from the face area management table 600, which was processed in step S1702 according to the priority coefficient 606.

[0140] In step S1802, the scene determination unit 407 calculates the number of face regions with an area ratio 604 that is greater than the face regions for which the area ratio threshold flag 607 is set to true in step S1704.

[0141] In step S1803, the scene determination unit 407 obtains the deviation of the distance variation of the face region obtained in step S1706.

[0142] In step S1804, the scene determination unit 407 obtains the deviation of the alignment of the face regions obtained in step S1708.

[0143] In steps S1805 and S1806, the scene determination unit 407 compares each value obtained in steps S1801 to S1804 with the corresponding item in the scene determination criterion table 1000 to determine the scene in which the image was taken.

[0144] As described above, according to this embodiment, without using personal information such as individual facial feature information, it is possible to determine which person in the image should be evaluated by estimating the scene in which the image was taken based on the area ratio and positional relationship of the facial regions of the people in the image. This makes it possible to evaluate the facial expressions of people in images containing multiple people more appropriately.

[0145] Furthermore, since there is no need to register reference facial images or characteristic information in advance, and there is no need to store and manage sensitive personal information, it leads to improved information management and avoidance of legal risks. In addition, the matching process of facial characteristic information is unnecessary, which reduces the load on the system.

[0146] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0147] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0148] The disclosures herein include the following information processing devices, information processing methods, and programs. [Item 1] A detection means that obtains location information of the face region of a person detected from an image, and obtains the area of ​​the face region from the location information, A determination means for determining the scene in which the image was taken based on the area of ​​the facial region and the positional relationship of the facial region, A determination means for determining the main subject in the image based on the scene determined by the determination means, An information processing apparatus characterized by having an evaluation means for evaluating facial expressions for the main subject determined by the determination means. [Item 2] The information processing apparatus according to item 1, characterized in that the detection means approximates the face region as a rectangular region and determines the area ratio of the face region to the whole image from the coordinate information of the rectangular region. [Item 3] The information processing apparatus according to item 2, further comprising analysis means for determining the difference in area ratios of preceding and succeeding face regions in order of the face region having the largest area ratio, and determining the deviation of the variation in the distance to the reference position of the face region and the deviation of the variation in alignment with respect to the direction in which the face regions are lined up. [Item 4] The information processing device according to item 3, characterized in that the reference position is the center or focus position of the image. [Item 5] The detection means sets a priority for the face region, The information processing apparatus according to item 3 or 4, characterized in that the analysis means calculates the difference in area ratios of preceding and succeeding face regions in order of the face region with the highest priority, and calculates the deviation of the distance variation of the face regions and the deviation of the alignment variation of the face regions. [Item 6] The information processing device according to item 5, characterized in that the detection means gives higher priority to face regions including the focus position of the image or face regions whose gaze direction is facing forward. [Item 7] The analysis means sets a threshold for the area ratio based on the difference, The information processing apparatus according to any one of items 3 to 6, characterized in that the determination means determines the number of facial regions in which the area ratio exceeds the threshold. [Item 8] The information processing device according to item 7, further comprising storage means for storing the positional information and priority of the face regions, the threshold of the area ratio, the variation deviation of the distance of the face regions, and the variation deviation of the alignment of the face regions as a table for each face region. [Item 9] The information processing apparatus according to item 8, characterized in that the determination means determines the scene in which the image was taken based on at least one of the area ratio of the largest face area, the number of face areas whose area ratio exceeds a threshold, the deviation of the variation in the distance of the face areas, and the deviation of the variation in the alignment of the face areas. [Item 10] The information processing device according to item 9, further comprising storage means for storing, as a table, judgment criteria that define, for each scene, the area ratio of the largest face region, the number of face regions whose area ratio exceeds a threshold, the deviation of the variation in the distance of the face regions, and the deviation of the variation in the alignment of the face regions. [Item 11] The information processing device according to item 10, characterized in that the determination means determines the scene by comparing the area ratio of the largest face region, the number of face regions whose area ratio exceeds a threshold, the deviation of the variation in the distance of the face regions, and the deviation of the variation in the alignment of the face regions with the determination criteria. [Item 12] The information processing device according to any one of items 9 to 11, characterized in that the aforementioned scene includes a portrait, a scene with a small number of people, a scene with a large number of people, a crowd scene, or a landscape scene. [Item 13] The information processing device according to item 12, characterized in that the evaluation means determines, for each scene, one of the following as the facial expression: smiling, closed eyes, blurred / out of focus, or not subject to evaluation. [Item 14] The information processing device according to item 13, characterized in that the evaluation means determines whether a smile is present in the portrait and the small group scene, but does not determine whether a smile is present in the large group scene, the crowd scene, or the landscape scene. [Item 15] The information processing device according to item 13, characterized in that the evaluation means determines whether the eyes are closed in the portrait, the small group scene, and the large group scene, but does not determine whether the eyes are closed in the crowd scene or the landscape scene. [Item 16] The information processing device according to item 13, characterized in that the evaluation means determines blur / blur in the portrait, the small group scene, and the large group scene, but does not determine blur / blur in the crowd scene or landscape scene. [Item 17] The information processing device according to any one of items 1 to 16, characterized in that the evaluation means excludes the image from evaluation if it is determined that a person is not the main subject. [Item 18] The information processing device according to item 14, characterized in that the evaluation means targets one or more face regions whose area ratio exceeds a threshold in the portrait scene, and determines a smile to be present in a predetermined proportion or more of the face regions whose area ratio exceeds a threshold in the small group scene. [Item 19] The information processing device according to item 15, characterized in that the evaluation means targets all face regions whose area ratio exceeds a threshold in the portrait scene, targets one or more face regions whose area ratio exceeds a threshold in the small group scene, and determines that the eyes are closed in all face regions in the large group scene. [Item 20] The information processing device according to item 16, characterized in that the evaluation means targets all face regions whose area ratio exceeds a threshold in the portrait scene, targets one or more face regions whose area ratio exceeds a threshold in the small group scene, and targets all face regions in the large group scene to determine blur / blur. [Item 21] An information processing method performed by an information processing device, A detection step involves obtaining location information of the face region of a person detected from an image, and obtaining the area of ​​the face region from the location information. A determination step of determining the scene in which the image was taken based on the area of ​​the facial region and the positional relationship of the facial region, A decision step to determine the main subject in the image based on the scene determined above, An information processing method characterized by comprising an evaluation step of evaluating the facial expressions of the main subject determined above. [Item 22] A program for causing a computer to function as one of the means of an information processing device described in any of items 1 through 20. [Explanation of symbols]

[0149] 101…Imaging device, 102…Information processing device, 103…Server device, 206…CPU, 400…Facial expression evaluation device, 405…Face position detection unit, 406…Face region information analysis unit, 407…Scene determination unit, 408…Evaluation target determination unit, 409…Image evaluation unit

Claims

1. A detection means that obtains location information of the face region of a person detected from an image, and obtains the area of ​​the face region from the location information, A determination means for determining the scene in which the image was taken based on the area of ​​the facial region and the positional relationship of the facial region, A determination means for determining the main subject in the image based on the scene determined by the determination means, An information processing apparatus characterized by having an evaluation means for evaluating facial expressions for the main subject determined by the determination means.

2. The information processing apparatus according to claim 1, characterized in that the detection means approximates the face region as a rectangular region and determines the area ratio of the face region to the whole image from the coordinate information of the rectangular region.

3. The information processing apparatus according to claim 2, further comprising analysis means for determining the difference in area ratios of preceding and succeeding face regions in order of the face region with the largest area ratio, and for determining the deviation of the variation in the distance to the reference position of the face region and the deviation of the variation in alignment with respect to the direction in which the face regions are lined up.

4. The information processing apparatus according to claim 3, characterized in that the reference position is the center or focus position of the image.

5. The detection means sets a priority for the face region, The information processing apparatus according to claim 3, characterized in that the analysis means calculates the difference in area ratios of preceding and succeeding face regions in order of the face region with the highest priority, and calculates the deviation of the distance variation of the face regions and the deviation of the alignment variation of the face regions.

6. The information processing apparatus according to claim 5, characterized in that the detection means gives higher priority to a face region including the focus position of the image or a face region whose gaze direction is facing forward.

7. The analysis means sets a threshold for the area ratio based on the difference, The information processing apparatus according to claim 3, wherein the determination means determines the number of facial regions in which the area ratio exceeds the threshold.

8. The information processing apparatus according to claim 7, further comprising storage means for storing the position information and priority of the face region, the threshold of the area ratio, the variation deviation of the distance of the face region, and the variation deviation of the alignment of the face region as a table for each face region.

9. The information processing apparatus according to claim 8, wherein the determination means determines the scene in which the image was taken based on at least one of the area ratio of the largest face area, the number of face areas whose area ratio exceeds a threshold, the deviation of the variation in the distance of the face areas, and the deviation of the variation in the alignment of the face areas.

10. The information processing device according to claim 9, further comprising storage means for storing, as a table, judgment criteria that define, for each scene, the area ratio of the largest face region, the number of face regions whose area ratio exceeds a threshold, the deviation of the variation in the distance of the face regions, and the deviation of the variation in the alignment of the face regions.

11. The information processing apparatus according to claim 10, characterized in that the determination means determines the scene by comparing the area ratio of the largest face region, the number of face regions whose area ratio exceeds a threshold, the deviation of the variation in the distance of the face regions, and the deviation of the variation in the alignment of the face regions with the determination criteria.

12. The information processing apparatus according to claim 9, characterized in that the aforementioned scene includes any of the following: a portrait, a scene of a small group, a scene of a large group, a crowd scene, or a landscape scene.

13. The information processing apparatus according to claim 12, characterized in that the evaluation means determines, for each scene, one of the following as the facial expression: smiling, closed eyes, blurred / out of focus, or not subject to evaluation.

14. The information processing device according to claim 13, characterized in that the evaluation means determines whether a smile is present in the portrait and the small group scene, but does not determine whether a smile is present in the large group scene, the crowd scene, or the landscape scene.

15. The information processing device according to claim 13, characterized in that the evaluation means determines whether the eyes are closed in the portrait, the small group scene, and the large group scene, but does not determine whether the eyes are closed in the crowd scene or the landscape scene.

16. The information processing device according to claim 13, characterized in that the evaluation means determines blur / blur in the portrait, the small group scene, and the large group scene, but does not determine blur / blur in the crowd scene or landscape scene.

17. The information processing apparatus according to claim 1, characterized in that the evaluation means excludes the image from evaluation if it is determined that a person is not the main subject of the image.

18. The information processing device according to claim 14, characterized in that the evaluation means targets one or more facial regions whose area ratio exceeds a threshold in the portrait scene, and targets a predetermined proportion or more of the facial regions whose area ratio exceeds a threshold in the small group scene to determine that a smile is present.

19. The information processing device according to claim 15, characterized in that the evaluation means targets all face regions whose area ratio exceeds a threshold in the portrait scene, targets one or more face regions whose area ratio exceeds a threshold in the small group scene, and determines whether the eyes are closed in all face regions in the large group scene.

20. The information processing device according to claim 16, characterized in that the evaluation means targets all face regions whose area ratio exceeds a threshold in the portrait scene, targets one or more face regions whose area ratio exceeds a threshold in the small group scene, and targets all face regions in the large group scene to determine blur / blur.

21. An information processing method performed by an information processing device, A detection step involves obtaining location information of the face region of a person detected from an image, and obtaining the area of ​​the face region from the location information. A determination step of determining the scene in which the image was taken based on the area of ​​the facial region and the positional relationship of the facial region, A decision step to determine the main subject in the image based on the scene determined above, An information processing method characterized by comprising an evaluation step of evaluating the facial expressions of the main subject determined above.

22. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 20.

Images