Information processing program, information processing method, and information processing device

Abstract

Proper installation of a camera device is important in AU estimation, because the prediction error, which is the magnitude of the error of an estimated value relative to the correct value for AU intensity, increases depending on the imaging conditions. In the present invention, a computer: acquires a first face image of a person; identifies a first state of an element of imaging conditions from the first face image; generates a second state of the element of the imaging conditions that has been modified so as to improve the first state; inputs the second state to a machine learning model generated from training by taking the state of the element of the imaging conditions of the face image as a feature quantity and taking errors in estimated values relative to the correct values for AU intensity as correct data, for respective AUs representing movement of facial expression muscles; estimates a prediction error for each of the AUs; determines whether all of the prediction errors for the respective AUs satisfy a predetermined criterion; and executes processing to identify an element of the imaging conditions that should be improved with respect to the first face image on the basis of the determination result as to whether the predetermined criterion is satisfied. The foregoing makes it possible to present a problem in the imaging conditions concerning a camera device for AU estimation.

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] Facial expressions play an important role in nonverbal communication. Facial expression estimation is an essential technology when developing computers that understand and support people. To estimate facial expressions, a method for describing facial expressions must first be defined. One known method for describing facial expressions is the AU (Action Unit). AUs are defined based on anatomical knowledge of facial muscles and represent facial movements involved in the expression of facial expressions. There are 44 types in total, and they are assigned a value from 0 to 5, with the lowest occurring intensity. Technologies for estimating AUs have also been proposed.

[0003] A typical form of AU estimation engine that estimates AUs is based on machine learning using a large amount of training data, and the training data used includes image data of facial expressions and the occurrence and intensity of each AU, which are the results of facial expression judgment.

[0004] Additionally, there is a need for a service that installs a camera device that captures images of faces, such as a webcam, and analyzes the captured images using AU estimation technology, for example, to quantitatively evaluate customer reactions or visualize the mental health status of employees. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-063172 Summary of the Invention [Problem to be solved by the invention]

[0006] However, for example, depending on the shooting conditions, such as brightness, distance from the camera device, and occlusion of the face, the prediction error, which is the size of the difference between the estimated value and the correct value of the AU strength, can increase in AU estimation, making it important to properly install the camera device.

[0007] In one aspect, the present invention aims to present the problem of the shooting conditions for the camera device for AU estimation. [Means for solving the problem]

[0008] In one aspect, the information processing program causes a computer to execute a process of acquiring a first facial image of a person, identifying a first state of an element of a shooting condition from the first facial image, generating a second state of the element of the shooting condition that has been changed to improve the first state, inputting the second state into a machine learning model generated by training, for each AU that represents the movement of facial muscles, the state of the element of the shooting condition of the facial image as a feature, and using the error of an estimated value of the AU strength relative to a correct value as correct data, estimating a prediction error for each AU, determining whether all of the prediction errors for each AU satisfy a predetermined criterion, and identifying an element of the shooting condition that should be improved for the first facial image based on the determination result of whether the predetermined criterion is satisfied. [Effects of the Invention]

[0009] In one aspect, the problem of the shooting conditions for the camera device for AU estimation can be presented. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of a problem with the conventional AU estimation technique. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of an information processing system according to this embodiment. [Figure 3] FIG. 3 is a block diagram showing an example of the configuration of the information processing device 10 according to this embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a method for training a prediction error estimation model according to this embodiment. [Figure 5] FIG. 5 is a diagram showing an example of a method for presenting elements of the shooting conditions according to this embodiment. [Figure 6] FIG. 6 is a diagram showing an example of presentation of elements of the shooting conditions according to this embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a method for determining a prediction error according to this embodiment. [Figure 8] FIG. 8 is a diagram showing an example of visualization of the determination process according to this embodiment. [Figure 9] FIG. 9 is a diagram showing an example of visualization of the imaging conditions that have affected the prediction error according to this embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of the flow of the training process according to this embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of the flow of the presentation process according to this embodiment. [Figure 12] FIG. 12 is a diagram showing an example of the hardware configuration of the information processing device 10 according to this embodiment. [Figure 13] FIG. 13 is a configuration diagram of an employee stress state analysis system. [Figure 14] FIG. 14 is a flowchart showing the operation of the stress state analysis system. [Figure 15] FIG. 15 is a diagram showing a customer facial expression analysis system in a retail store. [Figure 16] FIG. 16 is a flowchart showing the operation of the customer facial expression analysis system. DETAILED DESCRIPTION OF THE INVENTION

[0011] Below, examples of an information processing program, an information processing method, and an information processing device according to the present embodiment will be described in detail with reference to the accompanying drawings. Note that the present embodiment is not limited to these examples. Furthermore, each example can be appropriately combined within a consistent range.

[0012] Fig. 1 is a diagram showing an example of a problem with conventional AU estimation technology. Fig. 1 shows the state of elements of shooting conditions for an input image, which is a facial image of a person's face, and the allowable values ​​of the shooting conditions for each AU, targeting AU4 and AU12.

[0013] Here, the elements of the photographing conditions are, for example, feature quantities related to at least one of the distance between the person who is the subject and the camera, brightness or darkness, and occlusion of the upper and lower parts of the face, as shown in Fig. 1. Furthermore, the elements of the photographing conditions may also be, for example, feature quantities related to at least one of the resolution of the face image and the orientation of the face in the face image, but are not limited to these examples.

[0014] The distance from the camera, which is an element of the shooting conditions, is, for example, a feature related to resolution. Regarding the distance from the camera, for example, facial landmark detection is applied to the face image to detect the positions of both eyes, and the relationship between the distance between the both eyes and the distance from the camera is measured in advance, and based on this, the distance from the camera is calculated from the positions of both eyes.

[0015] Furthermore, the brightness or darkness of the elements of the photographing conditions is, for example, a feature amount related to lighting, and may be the average brightness value of pixels in the face image.

[0016] Furthermore, the occlusion of the upper and lower face, which is an element of the photographing conditions, may be, for example, a feature related to occlusion, and may be the ratio of the occluded area to the area of ​​the upper face / lower face. Furthermore, the occlusion of the upper and lower face is estimated using a machine learning model generated by training data consisting of, for example, "face images with occlusion of the upper face / lower face" and "the ratio of the occluded area to the area of ​​the upper face / lower face." Note that, instead of the unit of the upper face / lower face, a smaller area obtained by Voronoi division of 68 points generally defined as facial landmarks may also be used as the unit.

[0017] The state of the element of the imaging condition is, for example, a numerical value indicating the state of the element of the imaging condition. The allowable value of the imaging condition for each AU is, for example, a threshold value such as an upper or lower limit required to correctly estimate each AU. The allowable value is determined for each AU.

[0018] In the example of FIG. 1 , the state of the distance from the camera, which is an element of the shooting conditions, is indicated by state 201. Furthermore, the tolerance values ​​for the distance from the camera for AU4 and AU12 are indicated by tolerance values ​​202 and 203, respectively. Here, for example, AU4 is the movement of lowering the inner eyebrow, and AU12 is the movement of raising the corners of the mouth. In the example of FIG. 1 , the tolerance value is set as an upper limit, and it is determined that the AUs are correctly estimated if the state of the elements of the shooting conditions is within the tolerance value. Therefore, in the example of FIG. 1 , for the three elements of distance from the camera, brightness or darkness, and occlusion of the upper part of the face, both AU4 and AU12 are within the tolerance values. However, although the state 204 of occlusion of the lower part of the face is within tolerance value 205 for AU4, it exceeds tolerance value 206 for AU12, so there is a possibility that AU12 will not be correctly estimated.

[0019] Therefore, in the example of Figure 1, it is considered desirable to present occlusion of the lower part of the face as an element of the shooting conditions to be improved. However, for example, if shooting conditions to be improved for AU4 are presented, it is presented that there are no shooting conditions to be improved, while on the other hand, if shooting conditions to be improved for AU12 are presented, it is presented that the shooting condition to be improved is occlusion of the lower part of the face. Therefore, it is not sufficient to only present shooting conditions to be improved for a specific AU. Although only two types of AUs are listed in the example of Figure 1, in reality there are many more types of AUs, and it is necessary to present shooting conditions to be improved taking these types into consideration.

[0020] Therefore, one of the objectives of this embodiment is to present problems in the shooting conditions that should be improved for a plurality of AUs. Note that the plurality of AUs may be all AUs or only AUs for which estimation is performed.

[0021] Next, the configuration of an information processing system for presenting problems in shooting conditions that should be improved for multiple AUs will be described. Fig. 2 is a diagram showing an example of the configuration of an information processing system according to this embodiment. As shown in Fig. 2, the information processing system 1 is a system in which an information processing device 10 and a camera device 100 are connected via a network 50 so that they can communicate with each other.

[0022] The network 50 may be any of a variety of communication networks, whether wired or wireless, such as an intranet used within a facility where the target person is located. The network 50 may not be a single network, but may be configured, for example, such that an intranet and the Internet are connected via a network device such as a gateway or other device (not shown). If the information processing device 10 and the camera device 100 are directly connected, or if the information processing device 10 has a built-in camera function equivalent to that of the camera device 100, the network 50 need not be included in the information processing system 1.

[0023] The information processing device 10 may be, for example, a desktop PC (Personal Computer) or a notebook PC, or may be a mobile terminal such as a smartphone or a tablet PC.

[0024] The information processing device 10 acquires, for example, a video of a person's face captured by the camera device 100, detects the person's face included in the video using existing technology, and estimates the state of elements of the shooting conditions of the face image from the detected face image. Note that the existing technology may be, for example, YOLO (You Only Look Once), SSD (Single Shot Multibox Detector), or Faster R-CNN (Convolutional Neural Network) using deep learning.

[0025] Furthermore, the information processing device 10, for example, changes the state of the estimated elements of the shooting conditions so as to virtually improve them. Then, the information processing device 10, for example, inputs the virtually improved states of the elements of the shooting conditions into a machine learning model to estimate a prediction error for each AU. The machine learning model is generated by, for example, training each AU using the states of the elements of the shooting conditions of the face image as features and the error of the estimated value relative to the correct value of the intensity of the AU as correct data. Note that the training and generation of the machine learning model may be performed by the information processing device 10 or another information processing device.

[0026] Then, the information processing device 10 determines, for example, whether all of the prediction errors for each AU satisfy a predetermined standard, and identifies and presents elements of the photographing conditions that should be improved for the face image.

[0027] 2, the information processing device 10 is shown as a single computer, but may be, for example, a distributed computing system configured with multiple computers. Furthermore, the information processing device 10 may be a cloud computer device managed by a service provider that provides cloud computing services.

[0028] The camera device 100 is, for example, a camera for photographing a person. Images photographed by the camera device 100 are transmitted to the information processing device 10 at any time or at a predetermined timing. As described above, the camera device 100 may be a camera function built into the information processing device 10.

[0029] (Functional configuration of information processing device 10) Next, a functional configuration of the information processing device 10 that executes this embodiment will be described. Fig. 3 is a block diagram showing an example of the configuration of the information processing device 10 according to this embodiment. As shown in Fig. 3, the information processing device 10 has a communication unit 20, a storage unit 30, and a control unit 40.

[0030] The communication unit 20 is a processing unit that controls communication with other devices such as the camera device 100, and is, for example, a communication interface such as a network interface card or a USB interface.

[0031] The storage unit 30 has a function of storing various data and programs executed by the control unit 40, and is realized by a storage device such as a memory or a hard disk. The storage unit 13 stores, for example, image information 31, a prediction error estimation model 32, and an AU estimation model 33.

[0032] The image information 31 stores, for example, a video captured by the camera device 100, that is, a plurality of captured images that are a series of frames of a moving image.

[0033] The prediction error estimation model 32 stores, for example, information about a machine learning model for estimating, as a prediction error, the error of an estimated value of an AU intensity from a correct value based on the state of elements of the photographing conditions of a face image, and parameters for constructing the model. The machine learning model, i.e., the prediction error estimation model, is generated by machine learning, for example, using the state of elements of the photographing conditions of a face image as features and the error of an estimated value of an AU intensity from a correct value as a correct label. Note that the prediction error estimation model may be generated for each AU. Furthermore, the prediction error estimation model may be generated by the information processing device 10, or may be trained and generated by another information processing device.

[0034] The AU estimation model 33 stores, for example, information about a machine learning model for estimating the occurrence intensity of AUs from a facial image in which a person's face is captured, and model parameters for constructing the machine learning model. The machine learning model, i.e., the AU estimation model, is generated by machine learning using, for example, a facial image in which a person's face is captured by the camera device 100 as a feature and the occurrence intensity of AUs as a ground truth label. Note that the AU estimation model may be generated for each AU. Furthermore, the AU estimation model may be generated by the information processing device 10, or may be trained and generated by another information processing device.

[0035] The above information stored in the storage unit 30 is merely an example, and the storage unit 30 can store various information other than the above information.

[0036] The control unit 40 is a processing unit, such as a processor, that controls the entire information processing device 10. The control unit 40 includes a machine learning unit 41, an AU estimation unit 42, a shooting condition identification unit 43, and a prediction error estimation unit 44. Each processing unit is an example of an electronic circuit included in the processor or an example of a process executed by the processor.

[0037] The machine learning unit 41 trains and generates a prediction error estimation model, which is a machine learning model, using, for example, the state of elements of the shooting conditions of the face image as feature quantities for each AU and the error of the estimated value of the AU intensity from the correct value as correct data. Training the machine learning model can use, for example, a neural network, which is an existing technology. The correct data may be calculated based on the correct value and the estimated value of the AU intensity estimated from the face image by the AU estimation unit 42. Information about the generated prediction error estimation model is stored, for example, in the prediction error estimation model 32. When the training and generation of the machine learning model are performed by an information processing device other than the information processing device 10, the information processing device 10 does not need to include the machine learning unit 41.

[0038] Fig. 4 is a diagram showing an example of a method for training a prediction error estimation model according to this embodiment. As shown in Fig. 4, the machine learning unit 41 trains and generates a prediction error estimation model that, when inputting, for example, the state of elements of the shooting conditions, outputs a prediction error, which is the error of an estimated value relative to a correct value of the intensity of each AU. Note that in the example of Fig. 4, the prediction errors output from the prediction error estimation model only indicate errors for AU4 and AU12, but the prediction errors of the other AUs may also be output from the prediction error estimation model.

[0039] As shown in Fig. 4, for example, face images taken under various shooting conditions and correct values ​​of the intensities of each AU corresponding to each face image are prepared. In the example of Fig. 4, only correct values ​​of AU4 and AU12 are shown, but correct values ​​of the intensities of the other AUs are also prepared.

[0040] Then, for example, the state of each element of the shooting conditions is identified from each face image using existing technology by the shooting condition identification unit 43. Furthermore, the AU estimation unit 42 estimates the intensity of each AU from each face image using an AU estimation model. Furthermore, a prediction error is calculated between the estimated value of the intensity of each AU and the correct value of the intensity of each AU.

[0041] Then, the machine learning unit 41 performs training using, for example, the states of each element of the identified shooting conditions as feature amounts and the calculated prediction errors as correct answer data, and generates a prediction error estimation model.

[0042] The AU estimation unit 42 estimates the intensity of each AU from, for example, a video captured by the camera device 100, more specifically, a captured image. For example, the AU estimation unit 42 uses an AU estimation model to estimate AUs that indicate facial movements involved in expressing facial expressions, defined based on anatomical knowledge of facial muscles. The AU estimation model is a machine learning model generated by training using facial images as features and the intensity of AU occurrence as a ground truth label. Note that AU estimation may be performed for all 44 types or for each AU of the type to be estimated, and may be represented by a numerical value between 0 and 5. Furthermore, one facial expression may not necessarily correspond to one AU, but may correspond to multiple AUs.

[0043] The photographing condition specifying unit 43 specifies a first state of the element of the photographing condition from the first face image, for example, by using an existing technique. Also, the photographing condition specifying unit 43 generates a second state of the element of the photographing condition that is changed to improve the first state, for example.

[0044] Furthermore, the shooting condition specifying unit 43 determines whether all of the prediction errors for each AU estimated by the prediction error estimating unit 44 satisfy a predetermined criterion. Then, the shooting condition specifying unit 43 specifies elements of the shooting conditions to be improved for the first face image, for example, based on the determination result of whether all of the prediction errors for each AU satisfy the predetermined criterion. Note that the specified elements of the shooting conditions to be improved may be, for example, a combination of elements of the shooting conditions in which all of the prediction errors for each AU satisfy the predetermined criterion and the number of elements of the shooting conditions changed to improve them is the smallest. This will be described more specifically with reference to FIG. 5.

[0045] FIG. 5 is a diagram illustrating an example of a method for presenting elements of the photographing conditions according to this embodiment. As shown in FIG. 5, first, the photographing condition identification unit 43 virtually improves the state of some elements of the first state of the photographing condition elements identified from, for example, a face image, to generate a second state of the elements of the photographing conditions. As shown in FIG. 5, such improvement is performed based on patterns 1, 2, . . . , which improve at least one element of the first state of the elements of the photographing conditions. For convenience, the example of FIG. 5 also illustrates pattern 0, which does not improve the first state of the elements of the photographing conditions. Furthermore, while the example of FIG. 5 illustrates patterns that improve all combinations of the elements of the photographing conditions, elements of the photographing conditions that clearly do not require improvement may be excluded from the improvement targets using a threshold or the like.

[0046] Next, the shooting condition specification unit 43 calculates, for example, for each pattern, an acceptable value for the state of the element of the shooting condition at which the prediction error falls within a standard range using a prediction error estimation model, and compares it with a second state of the element of the shooting condition that has been changed to improve it.

[0047] Next, the imaging condition specifying unit 43 determines whether the second states of the elements of the imaging conditions changed to improve the imaging conditions are within the tolerance and satisfy the criteria for all target AUs, for example, based on a comparison result between the second states of the elements of the imaging conditions changed to improve the imaging conditions and the tolerance value. Then, for example, among the patterns satisfying the criteria, the elements of the imaging conditions having the smallest number of elements of the improved imaging conditions may be presented as the elements of the imaging conditions to be improved.

[0048] Furthermore, the photographing condition specification unit 43 presents, for example, the specified elements of the photographing conditions to be improved for the first face image. Fig. 6 is a diagram showing an example of presentation of elements of the photographing conditions according to this embodiment. Fig. 6 shows an example in which elements of the photographing conditions to be improved for the face image are presented in text form via a display device. However, the elements of the photographing conditions to be improved may be presented not only in text form but also in diagrams, decorations, etc., so as to be more visually easy to understand.

[0049] As shown on the left side of Fig. 6, the photographing condition specifying unit 43 presents elements of the photographing conditions that should be improved for the face image, i.e., occlusion of the lower part of the face in the example of Fig. 6. Furthermore, as shown on the right side of Fig. 6, while showing all elements of the photographing conditions, the elements of the photographing conditions that should be improved for the face image may be displayed in decorated text color.

[0050] In addition, the determination of whether the prediction error for each AU satisfies the criterion may be performed by comparing the prediction error output by inputting the second state into the prediction error estimation model for each AU with a preset criterion for the prediction error for each AU.

[0051] Fig. 7 is a diagram showing an example of a method for determining a prediction error according to this embodiment. The method for determining a prediction error shown in Fig. 7 is similar to the method shown in Fig. 5 up to the point where the state of some elements of a first state of elements of the shooting conditions identified from a face image is virtually improved using each pattern to generate a second state of the elements of the shooting conditions.

[0052] Next, the shooting condition specification unit 43 inputs, for example, the second state of the element of the shooting condition changed to improve the shooting condition for each pattern into the prediction error estimation model to calculate the prediction error, and compares the calculated prediction error with a preset prediction error standard. In Fig. 7, for example, the calculated prediction error is shown as prediction error 207, and the preset prediction error standard is shown as standard 208.

[0053] Next, the shooting condition specifying unit 43 determines whether or not all target AUs are within the calculated prediction error standard and satisfy the standard, for example, based on a comparison result between the calculated prediction error and a prediction error standard. Then, for example, among the patterns satisfying the standard, the shooting condition element with the smallest number of improved shooting condition elements may be presented as the shooting condition element to be improved.

[0054] Furthermore, the shooting condition specifying unit 43 may present, for example, in addition to the elements of the shooting conditions to be improved, a determination process such as the state of the elements of the shooting conditions that have been changed to improve them, the tolerance of the state of the elements of the shooting conditions, and the standard of the prediction error, which have been described using FIG. 5 and FIG. 7 .

[0055] Fig. 8 is a diagram showing an example of visualization of the determination process according to this embodiment. Fig. 8 shows an example in which elements of the shooting conditions to be improved for a face image are presented by text, figures, etc. via a display device. As shown on the left side of Fig. 8, in addition to the elements of the shooting conditions to be improved, the status of the elements of the shooting conditions that have been changed to improve them may also be presented.

[0056] Furthermore, as shown in the center of Fig. 8, in addition to the elements of the shooting conditions to be improved, the states of the elements of the shooting conditions that have been changed to improve them and the tolerance values ​​for the states of the elements of the shooting conditions may be presented. This is a presentation that is suitable for the case where the tolerance-based determination method described with reference to Fig. 5 is used.

[0057] Furthermore, as shown on the right side of Fig. 8, in addition to the elements of the shooting conditions to be improved, the prediction error calculated using the prediction error estimation model and the prediction error criterion may be presented. This presentation is suitable for the case where the prediction error-based determination method described with reference to Fig. 7 is used. Note that Fig. 8 only shows the presentation for AU4 and AU12, but the prediction error and the like may be presented for all AUs.

[0058] Furthermore, the shooting condition identification unit 43 may calculate and present, for example, the percentage of the magnitude of the influence that at least one element of the shooting conditions has on the prediction error. Fig. 9 is a diagram showing an example of visualization of the shooting conditions that have influenced the prediction error according to this embodiment. Fig. 9 shows an example in which the percentage of the magnitude of the influence that elements of the shooting conditions have on the prediction error is presented via a display device using text, drawings, and decorations.

[0059] As shown in Fig. 9, the ratio of the magnitude of the influence of the elements of the shooting conditions on the prediction error may be presented for each AU and element. Note that, although Fig. 9 shows presentation only for AU4 and AU12, the prediction error etc. may be presented for all AUs.

[0060] The ratio of the magnitude of the influence of the elements of the shooting conditions on the prediction error can be calculated, for example, as follows. First, the shooting condition specification unit 43 specifies the state of the elements of the shooting conditions from the input image, which is a face image, using, for example, an existing technology. Then, the shooting condition specification unit 43 executes, for example, the following process for each specified element i.

[0061] First, the shooting condition specification unit 43 inputs elements of the shooting condition state, in which the state of element i remains unchanged and the states of elements other than element i are optimized, into the prediction error estimation model to calculate a prediction error, and regards the prediction error as the prediction error based on the influence of element i alone. More specifically, for example, if element i is "occlusion of the lower part of the face," the state of "occlusion of the lower part of the face" is left unchanged, and "distance from camera" is set to 0, "brightness" is set to 0, and "occlusion of the upper part of the face" is set to 0, and these are input into the prediction error estimation model to obtain a prediction error. The prediction error is regarded as the prediction error based on the influence of "occlusion of the lower part of the face" alone. Furthermore, the proportion of the prediction error based on the influence of element i alone for each element i is regarded as the proportion of the magnitude of the influence on the prediction error. More specifically, for example, the ratio of the prediction error based on the influence of "occlusion of the lower part of the face" alone to the prediction error based on the influence of "distance from camera" alone, ..., is regarded as the proportion of the magnitude of the influence on the prediction error.

[0062] Returning to the explanation of Figure 3, the prediction error estimation unit 44 inputs the second state generated by the shooting condition identification unit 43 into the prediction error estimation model trained and generated by the machine learning unit 41 for each AU, and estimates the prediction error for each AU.

[0063] (Processing flow) Next, the flow of training processing for the prediction error estimation model according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the flow of training processing according to this embodiment. The training processing shown in Fig. 10 may be executed by the information processing device 10 or by another information processing device.

[0064] 10, the information processing device 10 acquires, for example, face images of a person's face captured under various shooting conditions by the camera device 100 from the image information 31 (step S101). At this time, the information processing device 10 may specify the state of the elements of the shooting conditions of the acquired face image using existing technology, or the state may be specified in advance, or a correct answer value may be prepared in advance and acquired.

[0065] Next, the information processing device 10 estimates the intensity of each AU from each face image acquired in step S101 using, for example, an AU estimation model (step S102). Note that an existing technology can be used to estimate the AU intensity in step S102.

[0066] Next, the information processing device 10 calculates a prediction error, which is correct data of the prediction error estimation model, for each face image, using the estimated value of each AU intensity estimated in step S102 and the correct value of each AU intensity (step S103). Regarding the calculation of the prediction error in step S103, for example, the absolute error between the estimated value of the AU intensity and the correct value can be calculated for each AU, and this can be used as the prediction error.

[0067] Next, the information processing device 10 trains and generates a prediction error estimation model for each face image, using the face image acquired in step S101 as features and the prediction error calculated in step S103 as correct answer data (step S104). Information about the generated prediction error estimation model is stored in the prediction error estimation model 32. After step S104 is executed, the training process shown in FIG. 10 ends.

[0068] Next, the flow of the process of presenting elements of shooting conditions to be improved according to this embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the flow of the presentation process according to this embodiment. The presentation process shown in Fig. 11 is executed by the information processing device 10, and may be executed, for example, at regular intervals or each time video is received from the camera device 100.

[0069] 11, the information processing device 10 first acquires, for example, a face image of a person's face captured by the camera device 100 from the image information 31 (step S201). In the presentation process shown in FIG. 11, the video captured by the camera device 100 is processed almost in real time, so the video is transmitted from the camera device 100 at any time and stored in the image information 31.

[0070] Next, the information processing device 10 identifies the first state of the element of the photographing condition from each face image acquired in step S201 by using, for example, an existing technique (step S202).

[0071] Next, the information processing device 10 virtually improves the first state of the identified element of the shooting condition to generate a second state of the element of the shooting condition (step S203). This may be performed based on each pattern for improving at least one element of the first state of the element of the shooting condition, as described with reference to FIG.

[0072] Next, the information processing device 10 inputs, for example, the second state of the element of the shooting condition generated in step S203 into a prediction error estimation model, and estimates a prediction error for each AU (step S204). The prediction error estimation model may be a machine learning model trained and generated by the training process shown in FIG.

[0073] Next, the information processing device 10 determines whether all of the prediction errors estimated in step S204 satisfy a predetermined standard (step S205). For example, as described with reference to FIG. 5, a prediction error estimation model is used to calculate an acceptable value for the state of the elements of the shooting conditions at which the prediction error falls within the standard, and the acceptable value is used as the standard. Alternatively, for example, assuming that multiple elements of the shooting conditions rarely deteriorate simultaneously, one element of the shooting conditions may be gradually deteriorated, and the state of the element when the prediction error exceeds the standard may be used as the acceptable value. Then, the state of the elements of the virtually improved shooting conditions is compared with the acceptable values ​​of all AUs, and if the state of the elements of the virtually improved shooting conditions falls within the acceptable values ​​of all AUs, it is determined that the prediction errors of all AUs satisfy the standard.

[0074] Next, the information processing device 10 identifies elements of the shooting conditions to be improved based on the determination result in step S205 (step S206). For example, a combination of elements whose prediction error satisfies the standard and has the smallest number of improved elements is identified as an element of the shooting conditions to be improved, that is, a problem.

[0075] Also, for example, if there are multiple combinations that satisfy the requirements of a combination of elements in which the prediction error satisfies the standard and the number of improved elements is the smallest, all or one of them is identified. Furthermore, when identifying one combination from multiple combinations, in order to make improvement easier, it is also possible to assign a degree of difficulty of improvement to the elements of the shooting conditions in advance, and identify the combination with the smallest total degree of difficulty.

[0076] More specifically, for each element, the difficulty level for improvement is set so that the greater the numerical value, the higher the difficulty level, for example, brightness: difficulty level 3, distance from camera: difficulty level 4, occlusion of upper face: difficulty level 2, and occlusion of lower face: difficulty level 2. For example, suppose that the combinations that satisfy the requirements are a first combination of "brightness, distance from camera" and a second combination of "occlusion of upper face, occlusion of lower face." In this case, the total difficulty level of the first combination is 3 + 4 = 7, and the total difficulty level of the second combination is 2 + 2 = 4. Since the second combination is smaller, "occlusion of upper face, occlusion of lower face" is identified as an element that needs improvement.

[0077] Next, the information processing device 10 presents, for example, the elements of the shooting conditions to be improved that have been identified in step S206 (step S207). After step S207 is executed, the presentation process shown in FIG.

[0078] (effect) As described above, the information processing device 10 acquires a first facial image of a person, identifies a first state of an element of the shooting conditions from the first facial image, generates a second state of the element of the shooting conditions that has been changed to improve the first state, and inputs the second state into a machine learning model generated by training, using the state of the element of the shooting conditions of the facial image as a feature for each AU that represents the movement of facial muscles, and the error of the estimated value of the AU strength relative to the correct value as correct data, to estimate a prediction error for each AU, determines whether all of the prediction errors for each AU satisfy a predetermined criterion, and identifies the element of the shooting conditions that should be improved for the first facial image based on the determination result of whether the predetermined criterion is satisfied.

[0079] In this way, the information processing device 10 virtually improves the shooting conditions estimated from the face image and estimates the prediction error for each AU under the improved shooting conditions. Then, the information processing device 10 identifies problems with the shooting conditions based on the determination result of whether all of the prediction errors satisfy the criteria. This allows the information processing device 10 to present problems with the shooting conditions for the camera device used for AU estimation.

[0080] In addition, the process executed by the information processing device 10 to identify elements of the shooting conditions to be improved for the first face image includes a process of identifying, based on the determination result of whether or not a predetermined criterion is satisfied, a combination of elements of the shooting conditions in which all of the prediction errors for each AU satisfy the predetermined criterion and the number of elements of the shooting conditions that have been changed to be improved is the smallest, as elements of the shooting conditions to be improved for the first face image.

[0081] This allows the information processing device 10 to more appropriately present problems with the shooting conditions for the camera device for AU estimation.

[0082] In addition, the process executed by the information processing device 10 to determine whether or not a predetermined criterion is met includes a process of calculating, using a machine learning model, an allowable value for the state of an element of the shooting conditions so that the prediction error falls within the predetermined criterion, and determining whether or not the predetermined criterion is met by comparing the allowable value with the second state.

[0083] This allows the information processing device 10 to more appropriately present problems with the shooting conditions for the camera device for AU estimation.

[0084] In addition, the process executed by the information processing device 10 to determine whether or not a predetermined criterion is satisfied includes a process to determine whether or not a predetermined criterion is satisfied by comparing, for each AU, the prediction error with a prediction error criterion for each AU.

[0085] This allows the information processing device 10 to more appropriately present problems with the shooting conditions for the camera device for AU estimation.

[0086] In addition, the process of identifying the first state executed by the information processing device 10 includes a process of identifying, as the first state, feature quantities related to at least one of face occlusion in the first face image, resolution of the first face image, lighting on the first face image, and face orientation.

[0087] This allows the information processing device 10 to more appropriately present problems with the shooting conditions for the camera device for AU estimation.

[0088] In addition, the process of identifying the first state executed by the information processing device 10 includes a process of identifying, as the first state, a feature related to the occlusion of a specific area of ​​the face in the first face image, as a feature related to the occlusion of the face.

[0089] This allows the information processing device 10 to more appropriately present problems with the shooting conditions for the camera device for AU estimation.

[0090] Furthermore, the information processing device 10 executes a process of presenting the identified elements of the photographing conditions to be improved for the first face image using at least one of text, graphics, and decorations.

[0091] This allows the information processing device 10 to present problems with the shooting conditions for the camera device for AU estimation in an easier-to-understand manner.

[0092] Furthermore, the information processing device 10 executes a process of presenting at least one of the first state, the second state, the prediction error, and a predetermined criterion.

[0093] This allows the information processing device 10 to present problems with the shooting conditions for the camera device for AU estimation in an easier-to-understand manner.

[0094] Furthermore, the information processing device 10 executes a process of calculating and presenting the ratio of the magnitude of the influence that at least one of the elements of the shooting conditions has on the prediction error, based on the second state.

[0095] This allows the information processing device 10 to present problems with the shooting conditions for the camera device for AU estimation in an easier-to-understand manner.

[0096] In addition, the information processing device 10 estimates the strength of the AU from the face image, calculates correct data based on the estimated value of the strength of the estimated AU and the correct value, and performs a process of training using elements of the shooting conditions of the face image as features and the calculated correct data to generate a machine learning model.

[0097] This allows the information processing device 10 to present problems with the shooting conditions for the camera device for AU estimation.

[0098] (system) The information, including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings, may be changed as desired unless otherwise specified. Furthermore, the specific examples, distributions, and numerical values ​​described in the embodiments are merely examples and may be changed as desired.

[0099] Furthermore, the specific form of distribution or integration of the components of each device is not limited to that shown in the figures. For example, the shooting condition identification unit 43 of the information processing device 10 in FIG. 3 may be distributed across multiple processing units, or the AU estimation unit 42 and the prediction error estimation unit 44 of the information processing device 10 may be integrated into a single processing unit. In other words, all or some of the components may be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions of each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0100] Fig. 12 is a diagram showing an example of the hardware configuration of an information processing device 10 according to this embodiment. As shown in Fig. 12, the information processing device 10 includes a communication interface 10a, an HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. The components shown in Fig. 12 are connected to each other via a bus or the like.

[0101] The communication interface 10a is a network interface card or the like, and communicates with other servers. The HDD 10b stores programs and a database (DB) that operate the processing units and functions of the information processing device 10 shown in FIG.

[0102] The processor 10d is a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), etc. The processor 10d may also be implemented by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor 10d is a hardware circuit that executes processes to realize the functions described in FIG. 3 and other figures by reading programs that execute the same processes as the processing units shown in FIG. 3 and other figures from the HDD 10b and expanding the programs in the memory 10c.

[0103] Furthermore, the information processing device 10 can also realize the same functions as those of the above embodiment by reading the program from a recording medium using a medium reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the information processing device 10. For example, the above embodiment may also be applied in the same way to cases where another information processing device executes the program or where the information processing device 10 and another information processing device cooperate to execute the program.

[0104] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and may be read out from the recording medium and executed by a computer.

[0105] This embodiment can also be applied to systems such as those described with reference to FIGS. 13 to 16. FIG. 13 is a configuration diagram of an employee stress state analysis system. The stress state analysis system shown in FIG. 13 is a system that acquires facial expressions of employees during a web conference captured by the internal camera of the employee's PC via a web conference system, and estimates the stress state from the facial expressions using existing technology. This enables, for example, a management department to identify employees with high stress levels and take appropriate measures. This embodiment can be applied to such a system for estimating stress states, and problems with the photographing conditions can be presented to employees during a web conference, for example.

[0106] Next, the flow of processing by the stress state analysis system shown in Fig. 13 will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the operation of the stress state analysis system. First, an employee presses the start button of the web conference system on their PC (step S301).

[0107] Next, the Web conference system starts the Web conference (step S302).

[0108] Next, the Web conference system starts transmitting images from the employee's camera to the stress state analysis system (step S303).

[0109] Next, the stress state analysis system starts estimating AU intensities based on the camera image and saving the estimation results (step S304).

[0110] The stress state analysis system also begins identifying problems with the shooting conditions based on the camera image (step S305). Note that the estimation of problems with the shooting conditions is performed by the process described with reference to Figures 5 and 11. Next, the stress state analysis system presents the problem on the PC each time a problem is identified (step S306). Alternatively, to reduce the annoyance of frequent presentations, the system may be configured to present the problem only when the same problem has been identified a predetermined number of times or more within a predetermined time period.

[0111] Next, the employee presses the end button of the Web conference system on the PC (step S307).

[0112] Next, the Web conference system ends the Web conference (step S308).

[0113] Next, the Web conference system ends the transmission of the images from the employee's camera (step S309).

[0114] Next, the stress state analysis system finishes estimating the AU intensity based on the camera image and saving the estimation result (step S310). Next, the stress state analysis system estimates the stress level based on the saved AU intensity (step S311). Next, if the estimated stress level is above a certain level, the stress state analysis system sends an alert to the management department by means of email or the like (step S312).

[0115] The stress state analysis system also ends the process of identifying problems with the shooting conditions based on the camera image (step S313). After steps S312 and S313 are executed, the operation of the stress state analysis system shown in FIG.

[0116] Another system to which this embodiment can be applied will now be described. Fig. 15 is a diagram showing a customer facial expression analysis system in a retail store. The customer facial expression analysis system shown in Fig. 15 is a system that, for example, in a retail store, captures customer facial expressions using multiple cameras installed in the store and analyzes differences in facial expressions depending on the area and time of day, in order to improve product placement, etc.

[0117] The customer facial expression analysis system applies this embodiment to present problems with the shooting conditions in each phase of "installation" and "post-installation" of the camera. For example, when installing the camera, the customer facial expression analysis system presents problems to the system installer in real time and urges them to install the camera appropriately. Furthermore, after installing the camera, the customer facial expression analysis system presents the system installer with frequently occurring problems based on camera footage of customer facial expressions captured over several days and urges them to adjust the installation appropriately.

[0118] Next, the flow of processing by the customer facial expression analysis system shown in Fig. 15 will be described with reference to Fig. 16. Fig. 16 is an operation flowchart of the customer facial expression analysis system. First, the system installer presses the start button of the customer facial expression analysis system (step S401).

[0119] Next, the customer facial expression analysis system starts acquiring images from multiple cameras, estimating AU intensities from the camera images, estimating facial expressions from the AU intensities, and saving the camera ID, the time of image acquisition, the customer ID, and facial expressions (step S402). Here, the facial expressions from the AU intensities are expressed, for example, by a level of comfort or discomfort. The camera ID and customer ID are, for example, identifiers that uniquely identify the camera and the customer, respectively.

[0120] Next, the customer facial expression analysis system starts identifying and saving problems with the shooting conditions for the images from the multiple cameras (step S403).

[0121] Next, the system installer presses the activation button for the installation mode of the customer facial expression analysis system (step S404).

[0122] Next, the customer facial expression analysis system starts presenting the problem on the display device each time a problem is identified (step S405).

[0123] Next, the system installer presses the end button for the installation mode of the customer facial expression analysis system (step S406).

[0124] Next, the customer facial expression analysis system ends presenting the problems on the display device each time a problem is identified (step S407).

[0125] Next, for example, after running the system for several days, the system installer presses the problem confirmation button for the customer facial expression analysis system (step S408).

[0126] Next, the customer facial expression analysis system tally and presents the number of occurrences of each problem during the operation period (step S409). The customer facial expression analysis system may also store camera footage and present it as specific problem cases.

[0127] Next, the system installer presses the stop button of the customer facial expression analysis system (step S410).

[0128] Next, the customer facial expression analysis system completes the process of acquiring multiple camera images, estimating AU intensities from the camera images, estimating facial expressions from the AU intensities, and saving the camera ID, the image acquisition time, the customer ID, and the facial expression (step S411).

[0129] Next, the customer facial expression analysis system ends the process of identifying and saving problems in the shooting conditions for the multiple camera images (step S412). After step S412 is executed, the operation of the customer facial expression analysis system shown in FIG. [Explanation of symbols]

[0130] 1. Information Processing Systems 10. Information processing equipment 10a communication interface 10b HDD 10c memory 10d processor 20 Communications Department 30 Storage section 31 Image information 32 Prediction Error Estimation Model 33 AU estimation model 40 Control Unit 41 Machine Learning Department 42 AU estimation part 43 Shooting condition specification unit 44 Prediction error estimation unit 50 Network 100 Camera Equipment 201 Status 202 Tolerance 203 Tolerance 204 Status 205 Tolerance 206 Tolerance 207 Forecast Error 208 Standards

Claims

1. obtaining a first facial image of the person; Identifying a first state of an element of a photographing condition from the first face image; generating a second state of the element of the imaging condition that has been modified to improve the first state; For each AU representing a movement of a facial muscle, the state of the element of the photographing condition of the face image is used as a feature, and the second state is input into a machine learning model generated by training using an error of an estimated value of the intensity of the AU relative to a correct value as correct data, thereby estimating a prediction error for each AU; determining whether all of the prediction errors for each AU satisfy a predetermined criterion; Identifying an element of the photographing conditions to be improved for the first face image based on a determination result of whether the predetermined criterion is satisfied. An information processing program that causes a computer to execute a process.

2. The process of identifying elements of the photographing conditions to be improved for the first face image includes: Based on the determination result of whether or not the predetermined criterion is satisfied, a combination of elements of the photographing conditions in which all of the prediction errors for each AU satisfy the predetermined criterion and the number of elements of the photographing conditions that have been changed to improve them is the smallest is identified as the elements of the photographing conditions that should be improved for the first face image.

2. The information processing program according to claim 1, further comprising:

3. The process of determining whether the predetermined criteria are met includes: Using the machine learning model, calculate tolerances for the states of the elements of the shooting conditions so that the prediction error falls within the predetermined standard; By comparing the tolerance value with the second state, it is determined whether the predetermined standard is satisfied.

2. The information processing program according to claim 1, further comprising:

4. The process of determining whether the predetermined criteria are met includes: For each AU, the prediction error is compared with a criterion for the prediction error for each AU to determine whether the predetermined criterion is satisfied.

2. The information processing program according to claim 1, further comprising:

5. The process of identifying the first state includes: As the first state, a feature amount relating to at least one of face occlusion in the first face image, resolution of the first face image, illumination for the first face image, and face orientation is identified.

2. The information processing program according to claim 1, further comprising:

6. The process of identifying the first state includes: As the feature amount related to the occlusion of the face, a feature amount related to the occlusion of a specific region of the face in the first face image is identified as the first state.

6. The information processing program according to claim 5, further comprising:

7. The information processing program according to claim 1, characterized in that the computer is caused to execute a process of presenting the identified elements of the shooting conditions that should be improved for the first facial image using at least one of text, figures, and decorations.

8. 2. The information processing program according to claim 1, wherein the program causes the computer to execute a process of presenting at least one of the first state, the second state, the prediction error, and the predetermined criterion.

9. 2. The information processing program according to claim 1, wherein the computer is caused to execute a process of calculating and presenting a proportion of the influence of at least one element of the shooting conditions on the prediction error based on the second state.

10. Estimating the intensity of the AU from the face image; Calculating the correct data based on the estimated value of the intensity of the estimated AU and the correct value; The machine learning model is generated by training using the elements of the photographing conditions of the face image as feature quantities and the calculated correct answer data.

2. The information processing program according to claim 1, wherein the information processing program causes the computer to execute processing.

11. obtaining a first facial image of the person; Identifying a first state of an element of a photographing condition from the first face image; generating a second state of the element of the imaging condition that has been modified to improve the first state; For each AU representing a movement of a facial muscle, the state of the element of the photographing condition of the face image is used as a feature, and the second state is input into a machine learning model generated by training using an error of an estimated value of the intensity of the AU relative to a correct value as correct data, thereby estimating a prediction error for each AU; determining whether all of the prediction errors for each AU satisfy a predetermined criterion; Identifying an element of the photographing conditions to be improved for the first face image based on a determination result of whether the predetermined criterion is satisfied. An information processing method characterized in that the processing is executed by a computer.

12. obtaining a first facial image of the person; Identifying a first state of an element of a photographing condition from the first face image; generating a second state of the element of the imaging condition that has been modified to improve the first state; For each AU representing a movement of a facial muscle, the state of the element of the photographing condition of the face image is used as a feature, and the second state is input into a machine learning model generated by training using an error of an estimated value of the intensity of the AU relative to a correct value as correct data, thereby estimating a prediction error for each AU; determining whether all of the prediction errors for each AU satisfy a predetermined criterion; Identifying an element of the photographing conditions to be improved for the first face image based on a determination result of whether the predetermined criterion is satisfied. An information processing device comprising a control unit that executes processing.

Images