Image data processing device and system

Abstract

This invention provides an image data processing apparatus and system capable of effectively detecting faces from images containing multiple faces. The processor performs the following processes: a first discrimination process, using a first discrimination model to detect human faces from an image represented by image data and discriminate the attributes of the detected faces; a clustering process, clustering regions in the image represented by image data where faces are detected according to the types of facial attributes; a selection process, selecting a second discrimination model suitable for a region based on the type information of facial attributes in the clustered regions; and a second discrimination process, using the second discrimination model suitable for the region to detect human faces from images in each region and discriminate the attributes of the detected faces.

Description

Technical Field

[0001] This invention relates to an image data processing apparatus and system, and more particularly to an image data processing apparatus and system for processing image data containing multiple faces. Background Technology

[0002] Patent Document 1 describes a technique that detects the faces of people in an image and determines the expressions of the detected faces, thereby estimating the state of each face.

[0003] Patent document 2 describes a technique that uses two detectors, one for detecting male faces and the other for detecting female faces, to determine gender while detecting faces.

[0004] Patent document 3 describes the following technique: by applying motion analysis to an image, regions with motion and regions without motion are separated, and under different conditions, the face is detected according to each separated region.

[0005] Patent document 4 describes a technique for recognizing the facial expressions of a specific user using a recognition model, which involves determining the tendency of the facial expressions of the user being identified, and selecting a recognition model for recognition based on the determination result.

[0006] Previous technical documents

[0007] Patent documents

[0008] Patent Document 1: Japanese Patent Application Publication No. 2019-219987

[0009] Patent Document 2: Japanese Patent Application Publication No. 2011-141799

[0010] Patent Document 3: Japanese Patent Application Publication No. 2018-142061

[0011] Patent Document 4: Japanese Patent Application Publication No. 2018-45350 Summary of the Invention

[0012] One embodiment of the present invention provides an image data processing apparatus and system capable of effectively detecting faces from an image containing multiple faces.

[0013] means for solving technical problems

[0014] (1) An image data processing apparatus for processing image data, the image data processing apparatus comprising: a memory; and a processor, the processor performing the following processing: a first discrimination processing, using a first discrimination model to detect a person's face from an image represented by the image data and discriminating the attributes of the detected face; a clustering processing, clustering regions in the image represented by the image data where the face is detected according to the types of facial attributes; a selection processing, selecting a second discrimination model applicable to the region based on the type information of facial attributes in the clustered region; and a second discrimination processing, using the second discrimination model applicable to the region to detect a person's face from an image of the region and discriminating the attributes of the detected face.

[0015] (2) The image data processing device according to (1), wherein,

[0016] A second discriminant model is prepared based on the types of attributes of multiple faces. In the clustering process, clustering is performed based on the types of attributes of the faces for which the second discriminant model has been prepared.

[0017] (3) The image data processing device according to (2), wherein,

[0018] A second discriminant model is prepared based on the types of facial attributes identified by the first discriminant model.

[0019] (4) The image data processing apparatus according to (2) or (3), wherein,

[0020] The second discriminant model detects faces with corresponding attributes with higher accuracy than the first discriminant model.

[0021] (5) An image data processing apparatus according to any one of (1) to (4), wherein,

[0022] The second discriminant model further classifies the categories and discriminates facial attributes in more detail within the scope of the corresponding category's attributes.

[0023] (6) An image data processing apparatus according to any one of (1) to (4), wherein,

[0024] The second discriminant model classifies and determines whether a face has the corresponding attribute.

[0025] (7) An image data processing apparatus according to any one of (1) to (6), wherein,

[0026] The processor performs each process once in the order of first discrimination processing, clustering processing, selection processing, and second discrimination processing.

[0027] (8) An image data processing apparatus according to any one of (1) to (6), wherein,

[0028] After the processor performs each process once in the order of first discrimination processing, clustering processing, selection processing, and second discrimination processing, it repeatedly performs clustering processing, selection processing, and second discrimination processing based on the result of the second discrimination processing.

[0029] (9) An image data processing apparatus according to any one of (1) to (8), wherein,

[0030] The first and second discriminant models, as facial attributes, at least distinguish facial expressions.

[0031] (10) The image data processing apparatus according to (9), wherein,

[0032] The first and / or second discriminant models are used as facial attributes to further determine the orientation of the face.

[0033] (11) An image data processing apparatus according to any one of (1) to (10), wherein,

[0034] The processor also performs an extraction process, in which regions containing people are extracted from the image represented by the image data as processing object regions, and clustering, selection, and second discrimination processes are performed on the extracted processing object regions.

[0035] (12) The image data processing apparatus according to (11), wherein,

[0036] In the extraction process, pathways are detected from the image, and the regions defined by the pathways are extracted as the processing target regions.

[0037] (13) An image data processing apparatus according to any one of (1) to (12), wherein,

[0038] The processor also performs map data generation processing, in which map data is generated by associating faces detected in the image represented by the image data with the positions and attributes within the image.

[0039] (14) The image data processing apparatus according to (13), wherein,

[0040] The processor also performs heatmap generation processing, in which heatmaps are generated based on map data.

[0041] (15) The image data processing apparatus according to (14), wherein,

[0042] The processor also performs display control processing, which causes the generated heatmap to be displayed on the monitor.

[0043] (16) The image data processing apparatus according to (14) or (15), wherein,

[0044] The processor also performs output control processing, in which the generated heatmap is output to the outside.

[0045] (17) An image data processing apparatus according to any one of (1) to (16), wherein,

[0046] The memory stores a first discrimination model and a second discrimination model.

[0047] (18) An image data processing system comprising:

[0048] A photographic apparatus; and an image data processing apparatus that inputs and processes image data captured by the photographic apparatus (1) to (17). Attached Figure Description

[0049] Figure 1 This is a block diagram illustrating an example of the hardware structure of an image data processing device.

[0050] Figure 2 This is a block diagram of the main functions of an image data processing device.

[0051] Figure 3 This is a diagram illustrating an example of facial detection results based on the first discriminant.

[0052] Figure 4 This is a concept diagram of clustering processing.

[0053] Figure 5 This is a concept map for region extraction.

[0054] Figure 6 This is the functional block diagram of the second discriminant.

[0055] Figure 7 This is a diagram illustrating an example of facial detection results based on the first detection unit.

[0056] Figure 8 This diagram illustrates an example of integrating detection results based on the detection result integration unit.

[0057] Figure 9 This is a flowchart illustrating the processing steps of a face detection action based on an image data processing device.

[0058] Figure 10 This is a flowchart illustrating the processing steps of the facial detection action based on a variant example.

[0059] Figure 11 This is a diagram illustrating an example of the relationship between facial attributes identified by the first discriminant model and the categories identified by the second discriminant model.

[0060] Figure 12 This is a diagram representing an example of the face detection results using the first discriminant model.

[0061] Figure 13 This is a diagram representing an example of the results of clustering.

[0062] Figure 14 This is another example of a region setting.

[0063] Figure 15 This is a diagram representing the general structure of an emotion measurement system.

[0064] Figure 16 This is a block diagram of the functions implemented by the image data processing device.

[0065] Figure 17 This is a diagram representing an example of a heatmap.

[0066] Figure 18 This is another example of a heatmap.

[0067] Figure 19 This is another example of a heatmap.

[0068] Figure 20 This is a conceptual diagram of facial detection processing.

[0069] Figure 21 This is a concept diagram representing an example of a captured image.

[0070] Figure 22 This is a concept graph representing an example of clustering within a single box.

[0071] Figure 23 This is a flowchart illustrating the processing steps of the face detection action.

[0072] Figure 24 This is a block diagram illustrating the main functions of an image data processing device capable of automatically extracting and processing the target area.

[0073] Figure 25 This is a concept diagram for processing object region extraction. Detailed Implementation

[0074] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0075] [First Implementation Method]

[0076] In events such as concerts, theatrical performances, and sporting events, measuring and collecting the emotions of attendees throughout the entire event allows for the analysis of various information related to the event. For example, in concerts, the collected information can be used to analyze the level of excitement for each piece. Furthermore, by establishing a correlation with location information within the venue and recording the emotional information of each attendee, the distribution of excitement states within the venue can be analyzed. Moreover, by determining the center of excitement based on the information on the distribution of excitement, it is possible to identify attendees who are experiencing heightened excitement.

[0077] The measurement of emotions is based on images taken inside the venue. Specifically, the emotions of each attendee are determined by detecting their faces and judging their expressions from the images taken inside the venue.

[0078] In recent years, by improving camera performance, it has become possible to capture hundreds or even thousands of attendees at a time in high resolution.

[0079] On the other hand, detecting individual faces and determining their expressions from images containing a large number of faces requires significant computational resources. Therefore, there is a problem of not being able to provide sufficient detection speed.

[0080] In this embodiment, an image data processing apparatus capable of effectively detecting faces and determining expressions even in images containing a large number of faces will be described. Expression is an example of a facial attribute.

[0081] [Device Structure]

[0082] Figure 1 This is a block diagram illustrating an example of the hardware structure of an image data processing device.

[0083] like Figure 1As shown, the image data processing apparatus 10 includes: a CPU (Central Processing Unit) 11, RAM (Random Access Memory) 12, ROM (Read Only Memory) 13, auxiliary storage device 14, an operation unit 15, a display 16, an input / output interface (I / F) 17, and a communication interface 18. That is, the image data processing apparatus 10 is configured as a computer and functions as an image data processing apparatus by executing a predetermined program through the computer. For example, a general-purpose computer such as a personal computer can be used as the computer that constitutes the image data processing apparatus 10. The CPU 11 is an example of a processor. The ROM 13 and / or the auxiliary storage device 14 store the program executed by the CPU 11 and various data. The ROM 13 and / or the auxiliary storage device 14 is an example of a memory. The auxiliary storage device 14 may employ an HDD (Hard Disk Drive), flash memory, SSD (Solid State Disk), etc. The operation unit 15 may consist of, for example, a keyboard, a mouse, or a touch panel. The display 16 may consist of, for example, a liquid crystal display (LCD) or an organic light emitting diode (OLED) display.

[0084] Figure 2 This is a block diagram of the main functions of an image data processing device.

[0085] like Figure 2 As shown, the image data processing apparatus 10 mainly includes an image data acquisition unit 21, a first discrimination unit 22, a clustering unit 23, a second discrimination model selection unit 24, a region extraction unit 25, a second discrimination unit 26, and a detection result output unit 27. The functions of each unit are achieved by the CPU 11, which acts as a processor, executing a predetermined program (image data processing program).

[0086] The image data acquisition unit 21 acquires image data that is to be processed. The image data is input to the image data processing device 10 via the input / output interface 17 or the communication interface 18.

[0087] The first discrimination unit 22 detects a person's face from the image represented by the image data acquired by the image data acquisition unit 21, and outputs information about the detected person's facial expression. Therefore, the image data input to the first discrimination unit 22 is the image data acquired by the image data acquisition unit 21. The position of the face within the image is determined, and the face is detected. Therefore, the first discrimination unit 22 outputs information about the position and expression of the detected face.

[0088] The location of a face within an image is determined, for example, by a bounding box. A bounding box is a rectangular box that encloses the detected face. The location of the bounding box is determined, for example, by the center coordinates of the bounding box.

[0089] Regarding facial expressions, one is selected from a set of pre-defined expressions. That is, the first discrimination unit 22 classifies and outputs the facial expressions of each character from the set of pre-defined expressions. In this embodiment, the facial expressions of each character are classified and output from three categories: "smiling," "angry," and "expressionless."

[0090] The first discrimination unit 22 uses a first discrimination model, which has been learned, to detect the faces of each person in the image and to discriminate the facial expressions of the detected persons (first discrimination processing). The first discrimination model generates images containing faces as learning data through machine learning. As the first discriminant model, for example, it can use object detection CNNs (Convolutional Neural Networks) such as R-CNN (Regions with Convolutional Neural Networks), Fast R-CNN (Fast Regions with Convolutional Neural Networks), Faster R-CNN (Faster Regions with Convolutional Neural Networks), Mask R-CNN (Mask Regions with Convolutional Neural Networks), SSD (Single Shot Multibox Detector), YOLO (You Only Look Once), YOLOv2 (You Only Look Once version 2), and YOLOv3 (You Only Look Once version 3). The first discriminant model is stored in ROM 13 or auxiliary storage device 14 as memory.

[0091] Typically, in facial expression recognition using a learned model, the accuracy (expression score) of each expression is output as the recognition result. The accuracy scores of each expression are output so that the total is 1. Therefore, by determining the expression with the highest accuracy, the detected facial expression can be identified. For example, when the facial expressions are classified as "smiling," "angry," and [no expression], if the accuracy scores for each expression are 0.98 for smiling, 0.01 for angry, and 0.01 for no expression, the facial expression is identified as "smiling."

[0092] Figure 3 This is a diagram showing an example of the detection results of a face based on the first discriminant.

[0093] exist Figure 3In the example shown, faces detected from image Im are represented by bounding boxes BB. Therefore, faces not surrounded by bounding boxes BB are faces that failed to be detected by the first discrimination unit 22.

[0094] In addition, Figure 3 In the diagram, the label L attached to each bounding box BB indicates the type of expression being identified for the detected face.

[0095] The clustering unit 23 performs clustering processing (clustering) on ​​the regions of detected faces within the image represented by the image data. In this embodiment, the regions of detected faces are clustered according to the type of the second discrimination model used in the second discrimination unit 26. For example, if a second discrimination model is prepared for each type of facial expression determined by the first discrimination unit 22, clustering is performed according to the type of facial expression determined by the first discrimination unit 22. In the image data processing apparatus 10 of this embodiment, a second discrimination model is prepared for each type of facial expression determined by the first discrimination unit 22. Therefore, in the image data processing apparatus 10 of this embodiment, clustering is performed according to the type of facial expression determined by the first discrimination unit 22. As described above, the first discrimination unit 22 determines three types of expressions: "smiling," "angry," and "expressionless." Therefore, the clustering unit 23 performs clustering according to each of these three types of expressions.

[0096] Figure 4 This is a concept diagram of clustering processing.

[0097] First, faces detected from the image are clustered according to the types of the second discrimination model. In this embodiment, since the second discrimination model is prepared according to each type of facial expression determined by the first discrimination unit 22, clustering is performed based on the types of facial expressions determined by the first discrimination unit 22. Figure 4 In the diagram, the face enclosed by the rectangular frame represents the face detected by the first discrimination unit 22. Furthermore, the label attached to each frame indicates the type of cluster it belongs to. Label F1 is the label attached to the face belonging to the cluster of smiling expressions. Label F2 is the label attached to the face belonging to the cluster of angry expressions. Label F3 is the label attached to the face belonging to the cluster of expressionless expressions.

[0098] Clustering employs well-known methods. For example, clustering algorithms such as k-means clustering are used.

[0099] After clustering, a region is defined for each cluster. Each region is defined as the area containing all faces belonging to that cluster. For example, it is defined by finding the smallest rectangular bounding box that contains all faces belonging to each cluster. This bounding box is defined as the area in the horizontal direction of the image (in...). Figure 4(in the x-direction) and longitudinal direction (in Figure 4 The region within the rectangle tangent to the outermost face (in the y-direction) is determined. The area within this rectangle is defined as the region belonging to each face cluster. Figure 4 In the diagram, the area represented by symbol Z1 is the area for a smiling expression. The area represented by symbol Z2 is the area for an angry expression. The area represented by symbol Z3 is the area for a neutral expression.

[0100] As described above, regions of the face detected in the image are clustered according to the type of the second discrimination model. In this embodiment, since the second discrimination model is prepared according to each type of facial expression determined by the first discrimination unit 22, clustering is performed according to the type of facial expression determined by the first discrimination unit 22.

[0101] The second discriminant model selection unit 24 performs a selection process (selection process) based on the facial expression type information of each clustered region, choosing a second discriminant model applicable to each region. The types of facial expressions clustered match the types of the second discriminant models. Therefore, the second discriminant model corresponding to the type of facial expression in each region is selected. Thus, for example, in... Figure 4 In the second discrimination unit 26, a second discrimination model for a smiling expression is selected in region Z1 (smiling expression). Similarly, a second discrimination model for an angry expression is selected in region Z2 (angry expression). Finally, a second discrimination model for a blank expression is selected in region Z3 (no expression). Information from the selected second discrimination model for each region is then applied to the second discrimination unit 26.

[0102] The region extraction unit 25 performs processing to extract images of each clustered region from the image data represented by the image data acquired by the image data acquisition unit 21. The region extraction unit 25 extracts images of each region based on the information of each region clustered by the clustering unit 23. Figure 5 This is a concept map for region extraction. Figure 5 This represents an example of extracting the smiley expression region Z1. For example... Figure 5 As shown, in the images of each region, images that occlude other regions are generated, and images of each region are extracted. Additionally, in... Figure 5 In the image, the areas indicated by diagonal lines are occluded areas. In the case of the image Iml of the smiling expression region Z1, the areas outside this region are occluded. The region extraction unit 25 extracts images corresponding to the number of clustered regions. The extracted images of each region are applied to the second discrimination unit 26.

[0103] The second discrimination unit 26 performs a process (second discrimination process) that detects the faces of people from the images of each clustered region and determines the facial expressions of the detected people. Therefore, the images of each region extracted by the region extraction unit 25 are input into the second discrimination unit 26. Similar to the first discrimination unit 22, faces are detected by determining their position within the image. Therefore, the second discrimination unit 26 outputs the position and expression of the detected face.

[0104] The second discriminant unit 26 uses multiple different second discriminant models to detect human faces in images and to discriminate the facial expressions of each detected person. Each second discriminant model is a learned model that generates images containing faces as training data through machine learning. In each second discriminant model, CNNs used for object detection, such as R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, SSD, YOLO, YOLOv2, and YOLOv3, are used.

[0105] Multiple second discrimination models are prepared for each type of facial expression. In this embodiment, a second discrimination model is prepared for each type of facial expression determined by the first discrimination unit 22. Therefore, a second discrimination model corresponding to a smiling expression (second discrimination model for smiling expression), a second discrimination model corresponding to an angry expression (second discrimination model for angry expression), and a second discrimination model corresponding to a blank expression (second discrimination model for blank expression) are prepared.

[0106] Each second discrimination model is composed of a discrimination model capable of detecting facial expressions of the corresponding type with higher accuracy than the first discrimination model. Specifically, the second discrimination model for a smiling expression is composed of a discrimination model that detects smiling faces from the input image with higher accuracy than the first discrimination model. Similarly, the second discrimination model for an angry expression is composed of a discrimination model that detects angry faces from the input image with higher accuracy than the first discrimination model. Furthermore, the second discrimination model for a blank expression is composed of a discrimination model that detects blank faces from the input image with higher accuracy than the first discrimination model.

[0107] The neural network structure of each second discriminant model is more multi-layered than that of the first discriminant model, enabling the detection of facial expressions corresponding to the first discriminant model with higher accuracy. Specifically, the hidden layers (intermediate layers) constituting the neural network are more multi-layered than those of the first discriminant model. Thus, by further multiplying the neural network structure (deep network structure), the second discriminant model can detect facial expressions of corresponding types with higher accuracy than the first discriminant model. On the other hand, compared to the second discriminant model, the first discriminant model can achieve a faster detection processing speed. That is, in the image data processing apparatus of this embodiment, the first discriminant model and the second discriminant model are in the following relationship: Although the first discriminant model has poor detection accuracy, it is constructed as a discriminant model with excellent detection processing speed. On the other hand, although the second discriminant model has poor detection processing speed, it is constructed as a discriminant model with excellent detection accuracy. As a result, in face detection using the first discriminant model, facial expressions of each type are automatically and randomly extracted from the image. On the other hand, in face detection using the second discriminant model, facial expressions corresponding to the second discriminant model are detected from the image with high accuracy.

[0108] The second discriminant model classifies whether an expression belongs to the corresponding category and determines the detected facial expression. For example, the second discriminant model for a smiling expression determines whether the detected facial expression is a smiling expression (whether it is a smiling expression or an expression other than a smiling expression). Similarly, the second discriminant model for an angry expression determines whether the detected facial expression is an angry expression (whether it is an angry expression or an expression other than an angry expression). Finally, the second discriminant model for a blank expression determines whether the detected facial expression is blank (whether it is blank or an expression other than a blank expression).

[0109] Figure 6 This is the functional block diagram of the second discriminant.

[0110] The second discrimination unit 26 mainly has the functions of the first detection unit 26a1, the second detection unit 26a2, the third detection unit 26a3, and the detection result integration unit 26b.

[0111] The first detection unit 26a1 uses a second discrimination model for smiling expressions to detect a person's face from the input image, and outputs the detected facial expression. Therefore, the first detection unit 26a1 inputs image data of the region of a smiling expression (image data from the image of the region of a smiling expression). As described above, it determines whether it is a smiling expression and outputs the expression.

[0112] Figure 7 This is a diagram showing an example of the detection results of the face based on the first detection unit.

[0113] like Figure 7As shown, faces are detected from the image of the smiling expression region Z1, and their expressions are determined. Since the second discrimination model can detect faces corresponding to the expressions with higher accuracy than the first discrimination model, it can detect faces that were not detected in the first discrimination model (see reference). Figure 3 ).

[0114] The second detection unit 26a2 uses a second discrimination model for angry expressions to detect a person's face from the input image, and then determines and outputs the detected facial expression. Therefore, the second detection unit 26a2 inputs image data of the region of the angry expression (image data of the image of the region of the angry expression). As described above, it determines whether it is an angry expression and outputs the expression.

[0115] The third detection unit 26a3 uses a second discrimination model for expressionless faces to detect a person's face from the input image, and outputs the detected facial expression. Therefore, image data of the expressionless region (image data of the image of the expressionless region) is input into the third detection unit 26a3. As described above, it determines whether the expression is expressionless and outputs an expression.

[0116] The test result integration unit 26b integrates the test results based on those from each testing unit. Figure 8 This diagram illustrates an example of integrating detection results based on the detection result integration unit. For example... Figure 8 As shown, the detection results (facial position and expression information) of the faces detected by each detection unit from the first detection unit 26a1 to the third detection unit 26a3 are integrated, and a facial detection result for one image is generated. The generated facial detection result corresponds to the facial detection result in the image represented by the image data acquired by the image data acquisition unit 21.

[0117] The detection result output unit 27 processes and outputs the detection results of the faces detected in a series of processes. The face detection result output includes information on the position and expression of each detected face.

[0118] The information of the facial detection results output from the detection result output unit 27 is associated with the image data and stored in the auxiliary storage device 14. Alternatively, it can be output to the outside via the input / output interface 17 and the communication interface 18. Furthermore, it can be output to the display 16 as needed. On the display 16, an image of the detected face surrounded by a bounding box and labeled with the determined expression is displayed (see reference). Figure 7 ).

[0119] [Detection Action]

[0120] Figure 9This is a flowchart illustrating the processing steps of a face detection action based on an image data processing device.

[0121] First, image data that is to be processed is acquired (step S1 / image data acquisition processing based on image data acquisition unit 21). The image data is input to image data processing device 10 via input / output interface 17 or communication interface 18.

[0122] Next, the acquired image data is processed to detect faces using a first discrimination model (step S2 / first discrimination processing based on the first discrimination unit 22). Faces are detected from the entire image represented by the image data. Furthermore, the position and expression within the image are determined and faces are detected. Face detection using the first discrimination model is a coarser detection than face detection using the second discrimination model. In other words, in face detection using the first discrimination model, faces with various expressions are automatically and randomly extracted from the image.

[0123] Next, the regions of faces detected from the image are clustered (step S3 / clustering processing based on clustering unit 23). Clustering is performed according to the types of the second discrimination model. In this embodiment, the second discrimination process is prepared according to each type of expression determined by the first discrimination model. Therefore, clustering is performed according to the types of expressions determined by the first discrimination model. Regions for each expression are defined by clustering. In this embodiment, regions for smiling expressions, angry expressions, and expressionless expressions are defined.

[0124] Next, a second discriminant model applicable to each clustered region is selected (step S4 / selection processing based on the second discriminant model selection unit 24). This processing is performed based on the facial expression type information of each clustered region. That is, a second discriminant model corresponding to the expression type for each region is selected. Therefore, a second discriminant model for smiling expressions is selected in regions with smiling expressions, a second discriminant model for angry expressions is selected in regions with angry expressions, and a second discriminant model for expressionless expressions is selected in regions with no expression.

[0125] Next, a second discrimination model applicable to each region is used to detect faces from the images of each region (step S5 / second discrimination processing based on the second discrimination unit 26). Therefore, faces are detected using the second discrimination model for smiling expressions in regions with smiling expressions, the second discrimination model for angry expressions in regions with angry expressions, and the second discrimination model for expressionless expressions in regions without expressions. In each region, faces are detected from the image of that region. Furthermore, the position and expression within the image are determined, and the face is detected. Regarding the corresponding expression, the second discrimination model can detect faces with higher accuracy than the first discrimination model. Therefore, the face detection here is more accurate than face detection using the first discrimination model. The face detection results for each region are integrated and output from the second discrimination unit 26.

[0126] The detection results of the faces output from the second discrimination unit 26 are used as the detection results of the faces in the input image and are output by the detection result output unit 27 (step S6). The detection result output includes information on the position and expression of each detected face.

[0127] As described above, in the image data processing apparatus 10 of this embodiment, firstly, a general first discrimination model is used to coarsely detect faces from the overall image. Next, the successfully detected face regions are clustered according to the type of a second discrimination model. Then, a second discrimination model specific to each clustered region is used to detect faces with high precision. Thus, faces can be effectively detected from images containing multiple faces.

[0128] The image data processing apparatus 10 of this embodiment is particularly effective in detecting the faces of attendees during events such as theater, concerts, and sports. That is, it is effective in detecting the faces of groups consisting of a large number of people. It is believed that during events such as theater, concerts, and sports, each attendee shares emotions with those around them in each scene and makes similar expressions. Therefore, by clustering expressions according to their types and selecting a discriminator (second discriminator) corresponding to each type to detect faces, even images containing a large number of faces can be detected with good accuracy. Thus, the image data processing apparatus 10 of this embodiment can effectively detect faces from images containing a large number of faces by utilizing the correlation of group emotions.

[0129] [Variation Example]

[0130] [A variation of the testing procedure]

[0131] In the above embodiment, each of the first discrimination process, clustering process, selection process, and second discrimination process is executed once in the order of the first discrimination process, clustering process, selection process, and second discrimination process. In this case, clustering is performed only based on the detection results of the face based on the first discrimination unit 22.

[0132] Face detection is also performed by the second discrimination unit 26. Therefore, clustering processing can also be performed using the face detection results based on the second discrimination unit 26. In this example, clustering processing is performed again using the face detection results based on the second discrimination unit 26, and face detection processing using the second discrimination model is performed. That is, after each processing step is performed once in the order of first discrimination processing, clustering processing, selection processing, and second discrimination processing, clustering processing, selection processing, and second discrimination processing are performed again based on the result of the second discrimination processing. In this example, clustering processing, selection processing, and second discrimination processing are performed repeatedly only for a predetermined number of times. That is, they are performed repeatedly until the predetermined number of times is reached. The predetermined number of times is an example of a specified condition.

[0133] Figure 10 This is a flowchart illustrating the processing steps of the detection action for a face based on a variant example.

[0134] First, image data that will be processed is acquired (step S11 / image data acquisition processing based on image data acquisition unit 21).

[0135] Next, the count n is set to 1. That is, n = 1 (step S12).

[0136] Next, the acquired image data is processed to detect faces using the first discrimination model (step S13 / first discrimination processing based on the first discrimination unit 22).

[0137] Next, the regions where faces were detected from the image are clustered (step S14 / clustering processing based on clustering unit 23). This clustering is based on the face detection results using the first discriminant model.

[0138] Next, a second discriminant model applicable to each clustered region is selected (step S15 / selection process based on the second discriminant model selection unit 24).

[0139] Next, a second discrimination model applicable to each region is used to detect faces from the images of each region (step S16 / second discrimination processing based on the second discrimination unit 26).

[0140] Next, count the count n. That is, set n = n + 1 (step S17).

[0141] Next, it is determined whether the count n exceeds the threshold (step S18). The threshold is a preset number of repetitions.

[0142] In step S18, if it is determined that the count n exceeds the threshold, the detection result of the face output from the second discrimination unit 26 is used as the detection result of the face for the input image and is output by the detection result output unit 27 (step S19).

[0143] In step S18, if it is determined that the count n has not exceeded the threshold, the process returns to step S14 for clustering. This clustering is based on the detection results of faces using the second discriminant model, thus including more faces. After clustering, based on the results, selection processing (step S15) and second discriminant processing (step S16) are performed. Clustering processing (step S14), selection processing (step S15), and second discriminant processing (step S16) are repeated until the count n exceeds the threshold.

[0144] Thus, by repeatedly performing clustering, selection, and second discriminant processing using the results of the second discriminant processing, more faces can be detected. That is, the second discriminant processing can detect faces that were not detected in the first discriminant processing. Therefore, by using these detection results and performing clustering and other processes again, even more faces can be detected.

[0145] Furthermore, in this example, a structure is set to repeatedly perform clustering, selection, and second discrimination processes a preset number of times, but the conditions for ending the repetition of each of these processes are not limited to this. For example, it can be set to repeatedly perform the processes until a preset number of faces are detected from the image. Moreover, the preset number of times can be set arbitrarily by the user. This setting is performed via the operation unit 15.

[0146] [A variation of the method for determining facial attributes]

[0147] In the above embodiment, the case of distinguishing three types of facial expressions was described as an example, but the types of facial expressions to be distinguished are not limited to this. For example, it is also possible to set the structure to distinguish facial expressions corresponding to seven types of emotions: "joy", "anger", "disgust", "surprise", "fear", "sadness" and "no emotion".

[0148] Furthermore, while the structure described above is designed to determine facial expressions as facial attributes, it can also be designed to determine other facial attributes such as age and gender, in addition to facial expressions. For example, it could also be designed to determine age and / or gender as facial attributes in addition to facial expressions.

[0149] Furthermore, it can be configured as a structure that determines facial orientation based on facial attributes. In this case, it can be configured as a structure that determines facial orientation in addition to facial expressions.

[0150] In this way, a structure can be set up to make a judgment based on multiple categories of facial attribute combinations.

[0151] Furthermore, the second discriminant model does not necessarily have to be prepared to correspond to all types of attributes that are discriminated by the first discriminant model.

[0152] Figure 11 This is a diagram illustrating an example of the relationship between facial attributes identified by the first discriminant model and the categories identified by the second discriminant model.

[0153] Figure 11 This example illustrates how the first discriminant model distinguishes facial expressions and orientations. In this example, facial expressions are categorized into three types: "smiling," "angry," and "expressionless." Furthermore, facial orientation is categorized into three directions: "facing forward," "right-lateral," and "left-lateral." Therefore, in this case, the first discriminant model distinguishes nine attributes of the face. Specifically, it distinguishes a smiling face facing forward (facing forward × smiling), a smiling face facing to the right (right-lateral × smiling), a smiling face facing to the left (left-lateral × smiling), an angry face facing forward (facing forward × angry), an angry face facing to the right (right-lateral × angry), an angry face facing to the left (left-lateral × angry), an expressionless face facing forward (facing forward × expressionless), an expressionless face facing to the right (right-lateral × expressionless), and an expressionless face facing to the left (left-lateral × expressionless).

[0154] The second discrimination model prepares discrimination models for smiling expressions, angry expressions, and expressionless expressions. Regarding the corresponding facial expressions, the discrimination models for each expression constituting the second discrimination model detect faces with higher accuracy than the first discrimination model. Therefore, it is possible to detect faces from multiple orientations for the corresponding facial expressions.

[0155] Figure 12 This is a diagram showing an example of the detection results for a face using the first discriminant model.

[0156] exist Figure 12In the image, faces enclosed by rectangular boxes are those detected from image Im. Each box contains a label indicating the facial attributes used to identify the face within the box. There are nine categories of labels (type of facial expression × type of facial orientation). The contents of each label are as follows: A1 is the label for a smiling face facing forward (front × smile). A2 is the label for a smiling face facing to the right (right horizontal × smile). A3 is the label for a smiling face facing to the left (left horizontal × smile). B1 is the label for an angry face facing forward (front × anger). B2 is the label for an angry face facing to the right (right horizontal × anger). B3 is the label for an angry face facing to the left (left horizontal × anger). C1 is the label for a neutral face facing forward (front × neutral). C2 is the label for a neutral face facing to the right (right horizontal × neutral). C3 is a label attached to a blank, expressionless face facing left (left horizontal × blank).

[0157] Figure 13 This is a diagram representing an example of the results of clustering.

[0158] Clustering is performed based on the types of expressions identified by the second discriminant model. Therefore, in this example, clustering is performed based on the type of facial expression, regardless of facial orientation. That is, clusters are created for smiling expressions, angry expressions, and expressionless expressions, and regions are defined for each expression. Figure 13 In the diagram, the region represented by symbol Z1 is the clustered region of smiling expressions. The region represented by symbol Z2 is the clustered region of angry expressions. The region represented by symbol Z3 is the clustered region of expressionless expressions.

[0159] Thus, when using the first discriminative model to detect faces, by increasing the number of facial attributes that can be discriminated, faces with multiple attributes can be detected. In this example, for each expression, faces with multiple orientations can be detected.

[0160] [A variation of face detection based on a second discriminant model]

[0161] In the above embodiment, the system is configured to determine whether a detected facial expression belongs to the corresponding category in the second discrimination model. The expressions determined by the second discrimination model are not limited to this. For example, it can be configured to further classify and discriminate facial attributes within the scope of attributes corresponding to the category. For example, as facial attributes, when discriminating facial expressions, it can be configured to further classify and discriminate expressions of the corresponding category in detail. In this case, for example, the second discrimination model corresponding to a laughing expression classifies a laughing expression as "big laugh," "medium laugh," "small laugh," etc. Thus, expressions can be detected in greater detail.

[0162] [A variation of clustering]

[0163] In the above embodiment, a bounding box containing the rectangles of faces belonging to each cluster is defined, and the region of each cluster is defined. The method for defining the region of each cluster is not limited thereto.

[0164] Figure 14 This is another example of a region setting.

[0165] exist Figure 14 In the example shown, for faces belonging to each cluster, a bounding box is set for the polygon tangent to the outermost face, and the area within this bounding box is set as the area of ​​each cluster. Additionally, in Figure 14 In the diagram, the area represented by symbol Z1 is the area for a smiling expression. The area represented by symbol Z2 is the area for an angry expression. The area represented by symbol Z3 is the area for a neutral expression.

[0166] In addition, for faces belonging to each cluster, a polygonal frame can be set by connecting the outermost face with a straight line, and the area within the frame can be set as the area of ​​each cluster.

[0167] [Regarding the processing of image data]

[0168] The image data to be processed can be either moving image data or still image data. Moving image data is processed frame by frame. In this case, it is not necessary to process all frames. A structure can be configured to process at pre-defined frame intervals.

[0169] [Second Implementation]

[0170] As mentioned above, by measuring and collecting the emotions of attendees at events such as concerts, theatrical performances, and sports events, it is possible to analyze a wide variety of information.

[0171] In this embodiment, a system for measuring the emotions of attendees during events such as concerts, theatrical performances, and sports activities (an emotion measurement system) will be described.

[0172] [System Architecture]

[0173] Figure 15 This is a diagram representing the general structure of an emotion measurement system.

[0174] The emotion measurement system 100 of this embodiment is configured to measure and collect the emotions of the audience (attendees) in a performance venue. The emotions of the audience are measured based on images captured from the audience.

[0175] like Figure 15 As shown, the emotion measurement system 100 of this embodiment includes: a camera device 110 for photographing audience members in a venue, and an image data processing device 120 for processing image data captured by the camera device 110. The emotion measurement system 100 of this embodiment is an example of an image data processing system.

[0176] The event venue 101 includes: a stage 103 for performers 102 to present their performances, and a viewing area V for audience members P to watch the performances. Seats 104 are arranged regularly in the viewing area V. Audience members P sit in seats 104 to watch the performances. The positions of each seat 104 are fixed.

[0177] [Photographic installation]

[0178] The photographic device 110 is composed of a digital camera capable of capturing moving images. In this embodiment, the entire viewing area V is photographed using a single photographic device 110. Therefore, the photographic device 110 is composed of a digital camera capable of photographing the entire viewing area V. Being capable of photographing the entire viewing area V means being able to include the entire viewing area V within the field of view and being able to capture images with image quality (resolution) sufficient to discern the facial expressions of each viewer present in the viewing area V from the captured image. The photographic device 110 photographs the viewing area V from a fixed position (fixed point).

[0179] [Image data processing device]

[0180] The image data processing device 120 inputs and processes image data output from the photography device 110, measures and records the emotions of each audience member in the venue. Emotions are determined from facial expressions. Therefore, in this embodiment, determining facial expressions means determining emotions. In this embodiment, three types of expressions are determined: "smiling," "angry," and "expressionless." In this case, a "smiling" expression corresponds to the emotion of joy or happiness. An "angry" expression corresponds to the emotion of anger. An "expressionless" expression corresponds to a state without a specific emotion. Map data is generated from the detected facial expressions (emotions) and recorded.

[0181] Furthermore, the image data processing device 120 generates a heat map representing the state or attributes of viewers at each location based on instructions from the user. The generated heat map is displayed on a monitor according to instructions from the user. It is also output to an external device according to instructions from the user.

[0182] The hardware structure of the image data processing apparatus 120 is substantially the same as that of the image data processing apparatus 10 of the first embodiment described above. That is, it is composed of a computer equipped with a CPU, ROM, RAM, auxiliary storage device, operation unit, display, input / output interface, and communication interface, etc. (see reference). Figure 1 ).

[0183] Image data captured by the camera device 110 is input to the image data processing device 120 via an input / output interface or a communication interface.

[0184] Figure 16 It is a block diagram of the functions implemented by the image data processing device.

[0185] like Figure 16 As shown, the image data processing apparatus 120 mainly includes a face detection unit 130, a map data generation unit 140, a heat map generation unit 150, a display control unit 160, and an output control unit 170. The functions of each unit are implemented by the CPU executing a predetermined program. The program executed by the CPU is stored in a ROM or auxiliary storage device, etc.

[0186] The face detection unit 130 detects faces from the image represented by the image data of the object being processed, and outputs the detected facial expressions. The functions of the face detection unit 130 are substantially the same as those of the image data processing apparatus 10 of the first embodiment described above. That is, the face detection unit has the functions of an image data acquisition unit, a first discrimination unit, a clustering unit, a second discrimination model selection unit, a region extraction unit, a second discrimination unit, and a detection result output unit. The functions of each unit are the same as those of the image data processing apparatus 10 of the first embodiment described above. Therefore, detailed descriptions are omitted.

[0187] The image data of the object being processed is image data output from the imaging device 110. The image data output from the imaging device 110 is image data of a moving image. The face detection unit 130 processes the image data frame by frame and detects faces from the image. It determines the position of the face within the image and detects the face. Therefore, the face detection unit 130 outputs information about the position and expression of the detected face frame by frame. In addition, detection does not necessarily have to be performed in all frames and can be performed at a preset frame interval.

[0188] The map data generation unit 140 performs map data generation processing (map data generation processing). The map data generation unit 140 generates map data based on the position and expression information of each face detected by the face detection unit 130. The map data is generated by associating and recording the position information of each face detected by the face detection unit 130 with the position information and determined expression information within the image. The map data is generated in frame units, in the same manner as face detection. The generated map data is associated with the image data of the generation source and recorded in the database 200. The database 200 is stored, for example, in an auxiliary storage device.

[0189] The heatmap generation unit 150 performs heatmap generation processing. Based on map data, the heatmap generation unit 150 generates heatmaps representing the state or attributes of each audience member. A heatmap is an image that displays the state or attributes of an audience member at each location using chroma or chroma intensity. In this embodiment, a heatmap is created from an image that displays the emotional (facial) state of an audience member at each location using chroma or chroma intensity.

[0190] Figure 17 This is a diagram representing an example of a heatmap.

[0191] exist Figure 17 In this method, a heat map is generated using the seating chart of the event venue. The seating chart is a planar representation of the seating arrangement in the event venue. The position of each seat corresponds to the position of each audience member. A one-to-one correspondence can be established between the position of each seat in the seating chart and the coordinate position of each audience member on the map data. Therefore, by displaying points at each seat location corresponding to the chromaticity or chromaticity intensity of the emotion (expression) of the audience member sitting in that seat, a heat map showing the emotional state (expression) of each audience member can be created. Figure 17 In the example shown, each expression—"laughing," "angry," "expressionless," and "uncertain"—is assigned a color saturation (in terms of density on the image) to represent the emotional (facial) state of the audience members in each seat. Furthermore, "uncertain" refers to audience members whose specific expressions could not be identified or whose faces could not be detected.

[0192] The data of the heat map generated by the heat map generation unit 150 is associated with the data of the generation source and recorded in the database 200.

[0193] The display control unit 160 performs a process (display control process) to display the data generated by the map data generation unit 140 on the display based on the display instructions input from the user via the operation unit. Furthermore, it displays the heat map generated by the heat map generation unit 150 on the display.

[0194] The output control unit 170 performs output control processing (output control processing) based on the output instruction from the user input via the operation unit, which outputs the data generated by the map data generation unit 140 to an external device. It also performs output control processing (output control processing) for the data of the heatmap generated by the heatmap generation unit 150 to an external device.

[0195] [effect]

[0196] In the emotion measurement system of this embodiment configured as described above, the audience's emotions are measured as follows.

[0197] First, the viewing area V is filmed using camera device 110. Filming is carried out throughout the entire duration of the event. That is, as moving images, filming is continuously performed from the beginning to the end of the event.

[0198] Image data captured by the photographic device 110 is input to the image data processing device 120. Image data can be input in real time or by summarizing data after the shooting is completed.

[0199] The image data processing apparatus 120 processes the input image data frame by frame and detects the viewer's face from each frame. It determines the position and expression within the image and detects the face. The processing for face detection is the same as that in the image data processing apparatus 10 of the first embodiment described above. Therefore, its details are omitted.

[0200] The image data processing device 120 generates map data in frame units based on the detection results of the face and records the generated map data in the database 200.

[0201] By processing all frames set as objects, the emotions of all audience members throughout all times of the event were measured.

[0202] The image data processing apparatus 120 generates a heatmap from the generated data according to instructions from the user. Furthermore, the image data processing apparatus 120 displays the generated heatmap on the display 16 according to instructions from the user. Finally, the image data processing apparatus 120 outputs the generated heatmap to an external device according to instructions from the user.

[0203] As described above, the emotion measurement system 100 according to this embodiment is able to measure and record the emotions of all audience members throughout the entire event.

[0204] [Variation Example]

[0205] [Examples of variations in photography]

[0206] In the above embodiments, it is set that the structure of the entire viewing area is captured by a single camera device. However, multiple camera devices can also be used to segment and capture images of the area to be viewed. In this case, it is possible to synthesize the images captured by each camera device to generate a single image, and detect facial structures from this single image. Alternatively, it is possible to detect facial structures separately from the images captured by each camera device. When detecting faces separately from the images captured by each camera device, the detection results are integrated or synthesized to obtain an overall processing result.

[0207] [Examples of map data variations]

[0208] Regarding map data, it is generated by associating and recording the location information and determined attribute information of each face detected in the image. Therefore, when determining age, gender, etc., in addition to facial expressions, this determined information is also associated and recorded.

[0209] [Examples of variations of heatmaps]

[0210] Heatmaps can be generated in various forms based on information recorded on map data. For example, map data may contain information about each viewer's attributes, including facial expressions, age, and / or gender. In this case, a heatmap displaying age and / or gender information can be generated. For example, an age heatmap might display points at each viewer's location and use a colorimetric scale corresponding to the age or age. Similarly, a gender heatmap might display points at each viewer's location and use a colorimetric scale corresponding to the gender. Finally, a heatmap combining both age and gender might display points at each viewer's location and use colorimetric scales corresponding to both age and gender.

[0211] Furthermore, heatmaps can be configured to visualize the structure of information estimated from information recorded in map data, rather than generating a structure that directly visualizes the information recorded in map data. For example, it is possible to estimate the excitement state of each audience member from the facial expression information recorded in map data and generate a heatmap of the estimated excitement state. In this case, the estimation process is performed by the CPU. Regarding the excitement state, for example, points are assigned corresponding to the facial expressions and quantified.

[0212] Furthermore, in the above embodiments, a heat map was generated using the seating chart of the event venue, but the form of the heat map is not limited to this. Figure 18 This is another example of a heatmap. In Figure 18 In the example shown, points are displayed at the locations of detected faces within the image, and the chromaticity of these points is displayed using chromaticity corresponding to emotions, thus generating a heatmap. Each point is displayed at a size corresponding to the dimensions of the faces detected from the image. For example, it is displayed as a circle inscribed in the bounding box. Figure 19 This is another example of a heatmap. In Figure 19 In the example shown, points are displayed on the image to generate a heatmap. The points are displayed overlapping at the locations of faces detected from the image and are displayed with a chromaticity corresponding to facial expressions (emotions).

[0213] [Third Implementation Method]

[0214] When a group of people exists, they often share similar emotions. However, when multiple groups exist, the dominant emotions vary from group to group. Therefore, it can be argued that clustering by group in the presence of multiple groups can more effectively detect faces.

[0215] In this embodiment, the process of clustering multiple groups to detect faces will be described. Furthermore, the basic structure of the image data processing apparatus is the same as that of the image data processing apparatus in the first embodiment described above. Therefore, only the different processing functions will be described here.

[0216] Figure 20 This is a conceptual diagram of facial detection processing.

[0217] Figure 20 This is a floor plan showing the event venue 101. Figure 20 In the event venue 101 shown, seats 104 are divided into six frames (frames BL1 to BL6) by aisle 105. In this case, clustering is performed separately for each frame. That is, in frame BL1, the regions of faces detected within frame BL1 are clustered. The other frames are similarly clustered for the regions of faces detected within the frames. The region constituting each frame is an example of the processing target region.

[0218] Figure 21 This is a concept diagram representing an example of a captured image.

[0219] When a camera is used to photograph the venue from a fixed position, the area occupied by each frame BL1 to BL6 within the captured image Im is uniquely determined. Therefore, the position, size, and shape of each frame BL1 to BL6 within the image Im can be determined in advance.

[0220] The clustering department performs clustering processing on each box based on the information (location, size, and shape) of the regions of each box BL1 to BL6 obtained in advance.

[0221] Figure 22 This is a concept graph representing an example of clustering within a single box. Figure 22 This represents an example of clustering in the second box BL2.

[0222] like Figure 22 As shown, clustering is performed on the faces detected within the bounding box. Additionally, Figure 22 The image shows an example of detecting facial expressions by distinguishing between three emotions: "smiling," "angry," and "expressionless," in the first discrimination unit. Furthermore, Figure 22The example shown illustrates a second discriminant model prepared with discriminant models corresponding to each of the expressions "laughing," "angry," and "expressionless." In this case, the expressions "laughing," "angry," and "expressionless" are clustered into three regions. Figure 22 In the diagram, the area represented by symbol Z1 is the region for a smiling expression. The area represented by symbol Z3 is the region for a neutral expression. Additionally, in... Figure 22 In the example shown, since no "angry" expression was detected, the regions were clustered into smiling and expressionless regions.

[0223] Thus, clustering is performed within the frames that form groups.

[0224] Figure 23 This is a flowchart illustrating the processing steps of a face detection action.

[0225] First, image data designated as the object to be processed is acquired (step S21). The image data is input to the image data processing device 10 via the input / output interface 17 or the communication interface 18.

[0226] Next, the acquired image data is processed to detect faces using the first discriminant model (step S22). Faces are detected from the overall image represented by the image data.

[0227] Next, within each frame, the regions where faces are detected are clustered (step S23). Clustering is performed based on the categories of the second discrimination model. Through clustering, regions for each expression are defined for each frame. In this embodiment, regions for smiling expressions, angry expressions, and expressionless expressions are defined for each frame.

[0228] Next, in each box, select the second discriminant model applicable to each region of the cluster (step S24).

[0229] Next, within each frame, a second discrimination model applicable to each region is used to detect faces from the image of each region (step S25). This process detects faces from each region of each frame. The information of faces detected from each region within each frame is integrated for each frame. Thus, faces are detected for each frame. By further integrating the information of faces detected for each frame, the face detection result for the entire image is output (step S26).

[0230] As described above, in the image data processing apparatus of this embodiment, clustering is performed on a per-frame basis. That is, clustering is performed on a per-group basis. Therefore, it is possible to more effectively detect faces by utilizing the emotional correlation within the groups.

[0231] [Variation Example]

[0232] [Variation 1 regarding the setting of the processing object region]

[0233] The processing target area within an image can be configured by the user. For example, it can be configured to overlay a frame onto an image captured by a photographic device, and define the processing target area within that frame. Thus, for example, in situations such as sporting events where the same image contains groups of spectators from different cheering teams, the processing target area can be appropriately distinguished.

[0234] [Example 2 regarding the setting of the processing object region]

[0235] The system can be configured to automatically extract regions containing people within an image and designate these extracted regions as processing target regions, thereby performing clustering processing on each processing target region. For example, it can be configured to automatically extract and process regions of frames BL1 to BL6 from a photographic image, as in the example of the above embodiment.

[0236] Figure 24 This is a block diagram of the main functions of an image data processing device that has the ability to automatically extract and process the target area.

[0237] like Figure 24 As shown, the image data processing apparatus 10 also has the function of a processing object area extraction unit 28.

[0238] The processing object region extraction unit 28 processes the image data acquired by the image data acquisition unit 21 and performs a process (extraction process) to automatically extract the region (processing object region) containing a person from the image represented by the image data.

[0239] Figure 25 This is a concept map for processing object region extraction.

[0240] like Figure 25 As shown, the processing object region extraction unit 28 extracts the regions that constitute straight lines from the image (in... Figure 25 In this process, the area represented by the diagonal line is used to define the region, and the region divided by the line is extracted as the processing target region. That is, the area forming the line is the region of path 105, and the region divided by path 105 is extracted as the processing target region. This process can be performed using known image recognition techniques.

[0241] Clustering unit 23 performs clustering processing on each processing object region extracted from the image.

[0242] [Other Implementation Methods]

[0243] In image data processing devices, the hardware structure of the processing units that perform various processes is implemented through various processors. These processors include: general-purpose processors that execute programs and function as various processing units, such as CPUs and / or GPUs (Graphics Processing Units); programmable logic devices (PLDs) such as FPGAs (Field Programmable Gate Arrays) whose circuit structures can be modified after manufacturing; and dedicated circuits such as ASICs (Application Specific Integrated Circuits) with circuit structures specifically designed to perform specific processes. The terms "program" and "software" have the same meaning.

[0244] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types. For example, a processing unit can be composed of multiple FPGAs, or a combination of a CPU and an FPGA. Furthermore, multiple processing units can also be composed of a single processor. As examples of multiple processing units composed of a single processor, firstly, in computers used in clients or servers, a combination of one or more CPUs and software constitutes a single processor, which performs the functions of multiple processing units. Secondly, in systems-on-chips (SoCs), a processor is used to implement the overall system functionality including multiple processing units using a single IC (Integrated Circuit) chip. Thus, various processing units are constructed using one or more of the aforementioned processors in a hardware structure.

[0245] Symbol Explanation

[0246] 10-Image data processing device, 11-CPU, 12-RAM, 13-ROM, 14-Auxiliary storage device, 15-Operation unit, 16-Display, 17-Input / output interface, 18-Communication interface, 21-Image data acquisition unit, 22-First discriminant unit, 23-Clustering unit, 24-Second discriminant model selection unit, 25-Region extraction unit, 26-Second discriminant unit, 26a1-First detection unit, 26a2-Second detection unit, 26a3-Third detection unit, 26b-Detection result integration unit, 27-Detection result output unit, 28-Processing object region extraction unit, 100-Emotion measurement system, 101-Event venue, 102-Performer, 103-Stage, 104-Seat 105-Passage, 110-Photography device, 120-Image data processing device, 130-Face detection unit, 140-Map data generation unit, 150-Heatmap generation unit, 160-Display control unit, 170-Output control unit, 200-Database, BB-Bounding box, Im-Image, Im1-Extracted image, L-Label, P-Viewer, V-Viewing area, BL1~BL6-Box, Z1-Clustered region of smiling expression, Z2-Clustered region of angry expression, Z3-Clustered region of expressionless expression, S1~S6-Processing steps for face detection actions, S11~S19-Processing steps for face detection actions, S21~S26-Processing steps for face detection actions.

Claims

1. An image data processing apparatus that processes image data, the image data processing apparatus comprising: a storage; and a processor, the processor performing: a first discrimination process that detects a face of a person from an image represented by the image data using a first discrimination model, and discriminates an attribute of the detected face; a clustering process that clusters regions in which a face is detected within the image represented by the image data according to a kind of the attribute of the face; a selection process that selects a second discrimination model suitable for the region based on kind information of the attribute of the face of the clustered region; and a second discrimination process that detects a face of a person from an image of the region using the second discrimination model suitable for the region, and discriminates an attribute of the detected face. 2.The image data processing apparatus according to claim 1, wherein the second discrimination model is prepared for each kind of the attribute of the face, in the clustering process, clustering is performed according to the kind of the attribute of the face for which the second discrimination model is prepared. 3.The image data processing apparatus according to claim 2, wherein the second discrimination model is prepared according to the kind of the attribute of the face discriminated by the first discrimination model. 4.The image data processing apparatus according to claim 2 or 3, wherein the second discrimination model detects a face of the corresponding kind of the attribute with higher accuracy than the first discrimination model. 5.The image data processing apparatus according to any one of claims 1 to 3, wherein the second discrimination model classifies the kind and discriminates the attribute of the face further in detail within the range of the corresponding kind of the attribute. 6.The image data processing apparatus according to any one of claims 1 to 3, wherein the second discrimination model classifies whether the face is of the corresponding kind of the attribute and discriminates the attribute of the face. 7.The image data processing apparatus according to any one of claims 1 to 3, wherein the processor performs each of the first discrimination process, the clustering process, the selection process, and the second discrimination process once in this order. 8.The image data processing apparatus according to any one of claims 1 to 3, wherein after the processor performs each of the first discrimination process, the clustering process, the selection process, and the second discrimination process once in this order, the clustering process, the selection process, and the second discrimination process are repeatedly performed based on the result of the second discrimination process. 9.The image data processing apparatus according to any one of claims 1 to 3, wherein the first discrimination model and the second discrimination model discriminate at least an expression as the attribute of the face. 10.The image data processing apparatus according to claim 9, wherein the first discrimination model and / or the second discrimination model further discriminate a direction of the face as the attribute of the face. 11.The image data processing apparatus according to any one of claims 1 to 3, wherein ​ The processor also executes extraction processing in which a region in which a person is present is extracted from an image represented by the image data as a processing target region, The processor executes the clustering processing, the selection processing, and the second discrimination processing on the extracted processing target region.

12. The image data processing apparatus according to claim 11, wherein In the extraction processing, a passageway is detected from the image, and a region divided by the passageway is extracted as the processing target region.

13. The image data processing apparatus according to any one of claims 1 to 3, wherein The processor also executes map data generation processing in which map data in which a face detected from an image represented by the image data is recorded in association with a position and an attribute in the image is generated.

14. The image data processing apparatus according to claim 13, wherein The processor also executes heat map generation processing in which a heat map is generated on the basis of the map data.

15. The image data processing apparatus according to claim 14, wherein The processor also executes display control processing in which the generated heat map is caused to be displayed on a display.

16. The image data processing apparatus according to claim 14, wherein The processor also executes output control processing in which the generated heat map is output to the outside.

17. The image data processing apparatus according to any one of claims 1 to 3, wherein In the memory, there are stored: The first discrimination model; and The second discrimination model.

18. An image data processing system comprising: a photographing device; and an image data processing apparatus according to any one of claims 1 to 17 that inputs and processes image data captured by the photographing device.

Images