Image data processing apparatus and image data processing system

Abstract

The present application provides an image data processing apparatus and an image data processing system capable of acquiring information of a person attribute of a person in a determined area with high precision. The image data processing apparatus performs the following processing: inputting a plurality of image data in which at least a part of a photographing range is repeated, detecting a face of a person in an image represented by each of the image data, identifying a person attribute of the person based on the detected face, generating, for each of the image data, map data in which the identified person attribute is recorded in correspondence with a position of the person in the image represented by the image data, interpolating the person attribute of the person repeated between the plurality of map data, and generating synthesis map data in which the plurality of map data after the interpolation is synthesized.

Description

Technical Field

[0001] This invention relates to an image data processing apparatus and an image data processing system, and more particularly to an image data processing apparatus and an image data processing system for processing image data acquired by multiple photographic devices. Background Technology

[0002] Patent document 1 describes the following technology: photographing an area with multiple spectators, obtaining information such as the facial expressions of each spectator through image recognition, and recording the obtained information in association with the location information of each spectator.

[0003] Patent document 2 describes a technique that visualizes information obtained by analyzing image data through color coding or the like, and displays the three-dimensional image data overlaid.

[0004] Previous technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2016-147011

[0007] Patent Document 2: Japanese Patent Application Publication No. 2017-182681 Summary of the Invention

[0008] The technical problem to be solved by the invention

[0009] One embodiment of the present invention provides an image data processing apparatus and an image data processing system capable of acquiring information on the attributes of a person within a defined area with high precision.

[0010] means for solving technical problems

[0011] (1) An image data processing apparatus for processing at least a portion of repeated image data within a photographic range acquired by a plurality of photographic devices, the image data processing apparatus comprising a processor that performs the following processing: detecting a face of a person in an image represented by each image data, and identifying a person's attributes based on the detected face; generating image data for each image data, recording the identified person's attributes in correspondence with the position of the person in the image represented by the image data; interpolating the person's attributes of repeated persons among the plurality of image data; and generating composite image data that synthesizes the interpolated image data.

[0012] (2) The image data processing apparatus according to (1), wherein,

[0013] The processor then performs further processing to generate a heatmap from the composite graph data.

[0014] (3) The image data processing apparatus according to (2), wherein,

[0015] The processor then performs the process of displaying the generated heatmap on the monitor.

[0016] (4) The image data processing apparatus according to (2) or (3), wherein,

[0017] The processor then performs further processing to output the generated heatmap to the outside.

[0018] (5) The image data processing apparatus according to any one of (1) to (4), wherein,

[0019] The processor checks the character attributes of characters that are repeated across multiple graph datasets. For character attributes that are missing in one graph dataset, it interpolates them using character attributes from another graph dataset.

[0020] (6) The image data processing apparatus according to any one of (1) to (5), wherein,

[0021] The processor calculates the recognition accuracy while recognizing a person's attributes.

[0022] (7) The image data processing apparatus according to (6), wherein,

[0023] The processor interpolates the attributes of duplicate people by replacing the attributes of people with relatively low recognition accuracy with the attributes of people with relatively high recognition accuracy.

[0024] (8) The image data processing apparatus according to (6), wherein,

[0025] The processor calculates the average of each person's attributes by assigning weights corresponding to the recognition accuracy, and then uses the calculated average to interpolate the attributes of duplicate persons.

[0026] (9) The image data processing apparatus according to any one of (6) to (8), wherein,

[0027] The processor has multiple recognition accuracies. It uses the information of the personal attributes of people with a recognition accuracies of more than the first threshold to interpolate the personal attributes of duplicate people.

[0028] (10) The image data processing apparatus according to any one of (6) to (9), wherein,

[0029] The processor further performs processing on the interpolated graph data, excluding information on person attributes with recognition accuracy below the second threshold.

[0030] (11) The image data processing apparatus according to any one of (1) to (9), wherein,

[0031] When it is impossible to interpolate the information of a person's attributes using other image data, the processor uses information of other person attributes that have similar temporal changes in the attribute information for interpolation.

[0032] (12) The image data processing apparatus according to any one of (1) to (11), wherein,

[0033] The processor then performs further processing to identify characters that are repeated across multiple graph datasets.

[0034] (13) The image data processing apparatus according to (12), wherein,

[0035] The processor determines which characters are repeated across multiple graph datasets based on the configuration relationships of characters in the graph data.

[0036] (14) The image data processing apparatus according to (12), wherein,

[0037] The processor determines which characters are repeated across multiple graph datasets based on their character attributes at each location in the graph data.

[0038] (15) The image data processing apparatus according to any one of (1) to (14), wherein,

[0039] The processor identifies at least one of the following as a person's attributes: gender, age, and emotion, based on the person's face.

[0040] (16) The image data processing apparatus according to any one of (1) to (15), wherein,

[0041] The processor instructs multiple photographic devices to photograph overlapping areas under different conditions.

[0042] (17) The image data processing apparatus according to (16), wherein,

[0043] The processor instructs multiple camera devices to photograph overlapping areas from different directions.

[0044] (18) The image data processing apparatus according to (16) or (17), wherein,

[0045] The processor instructs multiple photographic devices to capture overlapping areas of the photographic range with different exposures.

[0046] (19) An image data processing system comprising: a plurality of photographic devices that repeat at least a portion of a photographic range; and an image data processing apparatus for processing image data obtained by the plurality of photographic devices, wherein the image data processing apparatus includes a processor that performs the following processing: detecting a face of a person in an image represented by each image data, and identifying a person's attribute based on the detected face; generating image data for each image data that records the identified person's attribute in relation to the position of the person in the image represented by the image data; interpolating the person's attribute of a person that repeats among the plurality of image data; and generating composite image data that synthesizes the interpolated image data.

[0047] (20) The image data processing system according to (19), wherein,

[0048] Multiple photographic devices photographed overlapping areas under different conditions.

[0049] (21) The image data processing system according to (20), wherein,

[0050] Multiple photographic devices capture images of overlapping areas from different directions.

[0051] (22) The image data processing system according to (20) or (21), wherein,

[0052] Multiple photographic devices capture overlapping areas with different exposures. Attached Figure Description

[0053] Figure 1 This is a diagram representing the general structure of an image data processing system.

[0054] Figure 2 This is an example diagram showing the division of viewing areas.

[0055] Figure 3 This is a concept image of the area being filmed.

[0056] Figure 4 This is a block diagram illustrating an example of the hardware structure of an image data processing device.

[0057] Figure 5 This is a block diagram illustrating the functions implemented by the image data processing device.

[0058] Figure 6 This is a block diagram illustrating the functions of the image data processing unit.

[0059] Figure 7 This is a concept diagram of face detection.

[0060] Figure 8This is a block diagram illustrating the functions of the image data processing unit.

[0061] Figure 9 Conceptual graphs generated from graph data

[0062] Figure 10 This is a graph representing an example of graph data.

[0063] Figure 11 This is a diagram representing an instance of a database.

[0064] Figure 12 This is a concept diagram of face detection.

[0065] Figure 13 This is a concept diagram of face detection.

[0066] Figure 14 This is an example of a heatmap representing intensity.

[0067] Figure 15 This is an example of a display method for representing excitement levels.

[0068] Figure 16 This is a flowchart illustrating the image data processing steps in an image data processing system.

[0069] Figure 17 This is another example of a method for photographing the audience. Detailed Implementation

[0070] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0071] In events such as concerts and sporting events, the emotional information of all audience members in the venue is measured throughout the entire event, and various types of information can be analyzed by collecting this data. For example, in concerts, the level of excitement of the audience for each piece of music can be analyzed from the collected emotional information of all audience members. Furthermore, by recording and correlating the emotional information of each audience member with their location information, the distribution of excitement states within the venue can be analyzed. In addition, by determining the center of the excitement state distribution, it is possible to identify the audience members who are showing the most excitement.

[0072] When measuring emotional information of each audience member, image recognition technology can be used, for example. That is, image recognition can be used to infer the emotions of each audience member from images taken of them. The main method is to analyze facial expressions detected from the images.

[0073] However, it is difficult to measure the emotional information of all spectators in the venue without omission through image recognition. This is because faces may be obscured by obstacles (e.g., cheering flags, other spectators crossing in front, one's own or nearby spectators' hands, drinks, cameras, etc.), spectators may be facing away from their faces, and faces may be undetectable due to flashes and / or ghosting in the image (sunlight, reflections, flashes, etc.).

[0074] In this embodiment, a system is provided that can measure the emotional information of all spectators in the venue with high accuracy and without omission throughout the entire time of the event when measuring the emotional information of the audience through image recognition.

[0075] [System Composition]

[0076] This includes measuring the emotional information of all audience members at concert and other event venues, and the results collected will be used as an example for explanation.

[0077] Figure 1 This is a diagram showing the general structure of the image data processing system of this embodiment.

[0078] like Figure 1 As shown, the image data processing system 1 of this embodiment includes a spectator photography device 10 for photographing all spectators in the event venue and an image data processing device 100 for processing the image data photographed by the spectator photography device 10.

[0079] The event venue 2 has a stage 4 for performers 3 to present their performances and a viewing area V for the audience P to watch the performances. Seats 5 are arranged regularly in the viewing area V. The audience P sits in seats 5 to watch the performances. The positions of seats 5 are fixed.

[0080] [Audience Photography Installation]

[0081] The audience photography device 10 consists of multiple cameras C. Camera C is a digital camera capable of capturing moving images. Camera C is an example of a photography device. Audience member P is an example of a person photographed by the photography device.

[0082] The audience photography device 10 divides the viewing area V into multiple zones, and uses multiple cameras C to photograph each zone from multiple directions.

[0083] Figure 2 This is an example diagram illustrating the division of viewing areas. For example... Figure 2 As shown, in this example, the viewing area V is divided into 6 zones, V1 to V6. Each zone V1 to V6 is photographed individually from multiple directions by multiple cameras.

[0084] Figure 3 This is a concept image of the area being photographed. Figure 3 The image shows an example of the shooting area V1.

[0085] like Figure 3 As shown, in this embodiment, six cameras C1 to C6 photograph an area V1. Each camera C1 to C6 photographs area V1 from a pre-set position. That is, area V1 is photographed from a fixed point position. Each camera C1 to C6 is configured, for example, to be mounted on a remote-controlled gimbal (electric gimbal) and can adjust the shooting direction.

[0086] Camera 1 (C1) captures area V1 from the front. Camera 2 (C2) captures area V1 from the upper front. Camera 3 (C3) captures area V1 from the right. Camera 4 (C4) captures area V1 from the upper right. Camera 5 (C5) captures area V1 from the left. Camera 6 (C6) captures area V1 from the upper left. All cameras (C1-C6) shoot at the same frame rate and simultaneously.

[0087] The shooting range R1 to R6 of each camera C1 to C6 is set to cover area V1. Therefore, the shooting range R1 to R6 of each camera C1 to C6 overlaps with each other. Furthermore, each camera C1 to C6 is set so that each viewer is captured as approximately the same size within the captured image.

[0088] In this way, by shooting the area as the subject from multiple directions using multiple cameras, it is possible to effectively suppress the omission of the faces of the viewers within the area. For example, even if it is impossible to shoot with one camera due to obstacles, other cameras can still shoot, thus effectively suppressing missed shots.

[0089] Similarly, other areas V2 to V6 were also photographed from multiple directions by multiple cameras. Therefore, the number of cameras should match the number of areas to be divided.

[0090] Regarding the images captured by each camera C, it is required that the facial expressions of all viewers in the area being photographed be recognized. That is, a resolution capable of performing image recognition-based expression analysis is required. Therefore, it is preferable to use a high-resolution camera for the camera C constituting the viewer photography device 10.

[0091] Image data captured by each camera C is sent to the image data processing device 100. The image data sent by each camera C includes identification information for each camera C and information about the camera's shooting conditions. The information about the shooting conditions for each camera C includes information about the camera's location, shooting direction, and the date and time of the shooting.

[0092] [Image data processing device]

[0093] The image data processing device 100 processes image data transmitted by each camera C of the audience photography device 10, and measures the emotional information of each audience member within each image data. Furthermore, the image data processing device 100 generates [something] in each image data.

[0094] The image data is recorded by associating measured emotional information of each audience member with information about their position within the image. Furthermore, the image data processing apparatus 100 interpolates the image data generated from each image data. Finally, the image data processing apparatus 100 synthesizes the interpolated image data to generate composite image data representing the entire venue. Image data processing is performed frame by frame.

[0095] Furthermore, the image data processing apparatus 100 performs processing to visualize the composite image data as needed. Specifically, a heatmap is generated from the composite image data.

[0096] Figure 4 This is a block diagram illustrating an example of the hardware structure of an image data processing device.

[0097] The image data processing device 100 comprises a computer equipped with a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, HDD (Hard Disk Drive) 104, an operation unit 105, a display unit 106, and an input / output interface (I / F) 107. The CPU 101 is an example of a processor. The operation unit 105 may include, for example, a keyboard, a mouse, or a touch panel. The display unit 106 may include, for example, a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display.

[0098] Image data captured by each camera C of the audience photography device 10 is input to the image data processing device 100 through the input / output interface 107.

[0099] Figure 5 It is a block diagram of the functions implemented by the image data processing device.

[0100] like Figure 5As shown, the image data processing apparatus 100 mainly includes functions such as a photography control unit 110, an image data processing unit 120, an image data interpolation unit 130, an image data compositing unit 140, a data processing unit 150, a heat map generation unit 160, a display control unit 170, and an output control unit 180. The functions of each unit are implemented by the CPU 101 executing a predetermined program. The program executed by the CPU 101 is stored in ROM 103 or HDD 104. In addition to ROM 103 or HDD 104, the program may also be stored in flash memory or SSD (Solid State Disk).

[0101] The photography control unit 110 controls the operation of the audience photography device 10 based on the operation input from the operation unit 105. Each camera C constituting the audience photography device 10 performs shooting according to the instructions of the photography control unit 110. The control performed by the photography control unit 110 includes the control of the exposure of each camera C, the control of the shooting direction, etc.

[0102] The image data processing unit 120 generates image data from image data captured by each camera C of the viewer's photography device 10. Image data is generated for each image data.

[0103] Figure 6 This is a block diagram of the functions of the data processing department.

[0104] like Figure 6 As shown, the image data processing unit 120 mainly has functions such as a photographic information acquisition unit 120A, a face detection unit 120B, a person attribute recognition unit 120C, and a distribution generation unit 120D.

[0105] The photographic information acquisition unit 120A acquires photographic information from image data. Specifically, it acquires camera identification information and camera shooting conditions information contained in the image data. By acquiring this information, it is possible to determine the camera that captured the image data, and from which position and direction the image was taken of which area. Furthermore, it is possible to determine the date and time of the capture. The determined information is output to the distribution generation unit 120D.

[0106] The face detection unit 120B analyzes image data and detects the faces of people (spectators) in the image represented by the image data. Figure 7 This is a conceptual diagram of face detection. The face detection unit 120B detects faces by determining their location. The location of the face is determined by its coordinates (x, y) within the image Im. For example, the face detection unit 120B encloses the detected face with a rectangular box F, calculates the center coordinates of the box F, and determines the location of the face.

[0107] Furthermore, the technique for detecting human faces from images is a well-known technique, so details about it will be omitted.

[0108] Face detection is performed, for example, by scanning sequentially from the upper left to the lower right of image Im. Detected faces are numbered in the order of detection.

[0109] The character attribute recognition unit 120C identifies the character attributes of a person (audience member) based on an image of the person's (audience member's) face detected by the face detection unit 120B.

[0110] Figure 8 This is a conceptual diagram of the character attribute recognition process based on the character attribute recognition department.

[0111] In this embodiment, age, gender, and emotion are identified as personality attributes. Regarding techniques for identifying age, gender, and emotions from images, well-known techniques can be employed. For example, methods using image recognition models generated through machine learning, deep learning, etc., can be used.

[0112] For example, emotions can be identified from facial expressions. In this embodiment, facial expressions are categorized into seven types: "serious," "happy," "angry," "disgusted," "surprised," "fearful," and "sad," and the degree of each type is calculated to identify the emotion. The expressions "happy," "angry," "disgusted," "surprised," "fearful," and "sad" correspond to the emotions "happy," "angry," "disgusted," "surprised," "fearful," and "sad," respectively. "Serious" is expressionless and corresponds to a state without a specific emotion.

[0113] As a result of emotion recognition, an emotion score (emotion score) is output, which quantifies the degree of each emotion (emotion similarity). For example, the maximum value of the emotion score is set to 100. In this embodiment, the output is performed so that the sum of the scores for each emotion is 100.

[0114] Regarding age, it can be set to recognize the era rather than a specific age. For example, under 10 years old, teenagers, 20s, etc. In this embodiment, the era is recognized from the facial image. Regarding gender, male and female are identified from the facial image.

[0115] The distribution generation unit 120D generates image data based on the photographic information acquired by the photographic information acquisition unit 120A and the human attributes identified by the human attribute recognition unit 120C.

[0116] Image data is data recorded by associating the personal attributes of each audience member with the facial information of each audience member within the image. The position of an audience member is determined, for example, by the coordinates of their face.

[0117] Image data is generated for each image. Furthermore, photographic information from the source image data is appended to the image data. This includes identification information of the camera that captured the image data and information about the camera's shooting conditions. Therefore, it is possible to determine which region the image data belongs to, and also to what time the image data represents.

[0118] Figure 9 This is a concept map for generating graph data. Figure 9 The diagram shows an example of generating graph data from image data obtained by shooting region V1 with the first camera C1.

[0119] like Figure 9 As shown, the graph data is linked to the positions of each audience member within the image to record the audience member's personal attributes.

[0120] Figure 10 This is a graph representing an example of graph data.

[0121] like Figure 10 As shown, for each viewer whose face is detected from the image, information about their coordinate position and the attributes of the identified person is recorded. This graph data is generated for each image.

[0122] The graph data generated in the distribution generation unit 120D is recorded in the database 200.

[0123] Figure 11 This is a diagram representing an example of a database.

[0124] The graph data is recorded in the database in a time series, linking it to the information of cameras C1 to C6, which are its generation sources. Furthermore, the information of each camera C1 to C6 is also recorded in the graph data in a linking it to the information of the regions V1 to V6, which are the objects of the graph.

[0125] Database 200 manages image data generated by all cameras on an activity-based basis. Furthermore, database 200 also records image data interpolated by image data interpolation unit 130, composite image data generated from the interpolated image data, data obtained by processing composite image data, and heatmaps generated from the data obtained by processing composite image data. Database 200 is stored, for example, on HDD 104.

[0126] The graph data interpolation unit 130 interpolates the character attribute information of each audience member between graph data that repeatedly have the same audience member's character attribute information.

[0127] Image data that serves as the source of the image data contains repeated photographic ranges and information about the same viewer's character attributes within the overlapping areas of the photographic range.

[0128] The data in each image may not necessarily contain information on the attributes of all viewers. This is because the image data that serves as the source of the images may contain instances where faces cannot be detected or where facial attributes cannot be identified.

[0129] In the image data processing system of this embodiment, multiple cameras capture images of overlapping areas from multiple directions. Therefore, for example, sometimes even if it is impossible to capture a viewer's face with one camera, it can be captured by another camera.

[0130] In the image data processing system of this embodiment, the character attribute information of each audience member is interpolated between image data that repeatedly contain character attribute information of the same audience member. This generates high-precision image data.

[0131] The following describes the interpolation processing of the graph data performed by the graph data interpolation unit 130.

[0132] Figure 12 and Figure 13 This is an example of the detection results for a face. Figure 12 The image shown is an example of detecting a face from an image obtained when the first camera C1 captures region V1. Figure 13 The image shown illustrates an example of face detection from an image obtained when region V1 is captured by the second camera C2. Figure 12 and Figure 13 In the image, white circles represent the locations of viewers whose faces were detected. Conversely, black circles represent the locations of viewers whose faces were not detected.

[0133] The image data generated from the image captured by the first camera C1 is set as the first image data, and the image data generated from the image captured by the second camera C2 is set as the second image data.

[0134] like Figure 12 As shown, the faces of spectators P34, P55, P84, and P89 cannot be detected in the images captured by the first camera C1. Therefore, in this case, information regarding the character attributes of spectators P34, P55, P84, and P89 is missing in the data of the first image.

[0135] On the other hand, such as Figure 13 As shown, the faces of spectators P34, P55, P84, and P89 cannot be detected from the image captured by the second camera C2. Therefore, the second image data contains information about the character attributes of these spectators P34, P55, P84, and P89. In this case, the spectator information missing in the first image data can be interpolated from the second image data. That is, the character attribute information of spectators P34, P55, P84, and P89 missing in the first image data can be interpolated from the information in the second image data.

[0136] Similarly, as Figure 13 As shown, the faces of spectators P29, P47, P62, and P86 cannot be detected in the images captured by the second camera C2. Therefore, in this case, information regarding the character attributes of spectators P29, P47, P62, and P86 is missing from the data in Figure 2.

[0137] On the other hand, such as Figure 12 As shown, the faces of spectators P29, P47, P62, and P86 cannot be detected in the image captured by the first camera C1. Therefore, the character attribute information of these spectators P29, P47, P62, and P86 exists in the first image data. In this case, the missing spectator information in the second image data can be interpolated from the first image data. That is, the missing character attribute information of spectators P29, P47, P62, and P86 in the second image data can be interpolated from the information in the first image data.

[0138] Thus, the graph data generated from images with repeating regions contains information about the same audience's character attributes in the repeating regions of the image. Therefore, interpolation is possible when these attributes are missing.

[0139] Furthermore, in the above example, although an example of interpolating the information of the audience's character attributes that is missing between two graph data was explained, the information of each audience's character attributes was also interpolated between graph data that repeatedly have the same audience's character attribute information.

[0140] Regarding interpolation, the data is first compared between graph datasets that repeatedly contain the same audience member's attribute information to identify any missing audience member attribute information in each graph dataset. For graph datasets with missing audience member attribute information, interpolation is performed using the corresponding audience member attribute information from other graph datasets. When the same audience member's attribute information exists in multiple graph datasets, for example, information with high recognition accuracy is used.

[0141] When verifying data, data matching is performed based on the configuration relationships of each audience member. That is, duplicate audience members are identified by analyzing the positioning patterns of each audience member within the image. Furthermore, data matching can also be performed based on the personal attributes of audience members at each location.

[0142] The graph data that has undergone interpolation processing by the graph data interpolation unit 130 is recorded in the database 200 (see reference). Figure 11 ).

[0143] The image data synthesis unit 140 synthesizes the interpolated image data and generates a composite image data. This composite image data is a record that associates the personal attributes of all spectators in the venue with the position of each spectator's face.

[0144] Composite image data is generated from image data taken at the same time. Therefore, composite image data is generated sequentially according to a time series.

[0145] During compositing, information from the cameras is utilized. That is, image data is generated from images captured by the cameras. Since each camera captures a pre-defined area under pre-set conditions (position and orientation), compositing can be easily performed by utilizing its information.

[0146] Furthermore, the compositing process can also utilize image data that serves as the source for generating the graph data. That is, since image data and graph data correspond, graph data can be synthesized by compositing image data. For example, panoramic compositing methods can be used when compositing image data.

[0147] Thus, in the image data processing system 1 of this embodiment, multiple image data are synthesized, and a single composite image data is generated from the generated multiple image data. Therefore, even in large event venues, it is easy to generate image data that records the attributes of all attendees. Furthermore, even in small event venues, image data for the entire venue can be generated more efficiently than when generating image data by capturing the entire venue with a single camera. That is, because processing is performed in multiple areas, distributed processing is possible, and image data for the entire venue can be generated efficiently.

[0148] The generated composite graph data is linked to the source graph data and recorded in database 200 (reference). Figure 11 ).

[0149] The data processing unit 150 generates data about each audience member in the venue by processing the composite image data. The method of data generation depends on the user's settings. For example, it may generate data representing the emotional state of each audience member, data representing the emotional intensity of a specific emotion, or data representing the level of excitement.

[0150] Emotional state data can be obtained, for example, by extracting the highest-scoring emotion from the emotion recognition results. For instance, if a viewer's emotion recognition results (scores) are: serious: 12, happy: 75, angry: 0, disgust: 0, surprised: 10, fear: 3, sad: 0, the emotional state is happy.

[0151] Data representing the magnitude of a given emotion is data that quantifies the level or range of a given emotion.

[0152] The data on emotion levels are derived from emotion scores. For example, the data for the emotion level of happiness is obtained from the happiness score. Furthermore, the data for the emotion levels of happiness and surprise, for example, are obtained by summing the happiness and surprise scores. In this case, each emotion can be assigned a weight to calculate the emotion level data. That is, the score of each emotion can be set as a component of multiplying by a pre-defined coefficient and calculating the sum.

[0153] Data on the magnitude of emotions can be obtained, for example, by calculating the difference in emotion scores at predetermined time intervals. For instance, the magnitude of the emotion of happiness can be obtained by calculating the difference in happiness scores at predetermined time intervals. Furthermore, the magnitude of the emotions of happiness and sadness can be obtained, for instance, by calculating the difference between the happiness score and the sadness score at predetermined time intervals (e.g., the difference between the happiness score at time t and the sadness score at time t+Δt).

[0154] Regarding the magnitude of emotions, which emotion is chosen as the subject of measurement depends on the type of activity. For example, it is believed that the level of happiness during a concert primarily relates to audience satisfaction. Therefore, in the case of a concert, the level of happiness is used as the subject of measurement. On the other hand, it is believed that the magnitude of emotional amplitude (e.g., the amplitude of happiness versus sadness) primarily relates to audience satisfaction during watching a sporting event. Therefore, the magnitude of emotional amplitude is used as the subject of measurement during watching a sporting event.

[0155] Excitement level is a numerical value representing the degree of excitement experienced by each audience member. It is calculated using a pre-defined formula based on an emotion score. The formula Fn is defined as follows: for example, if the score for seriousness is S1, happiness is S2, anger is S3, disgust is S4, surprise is S5, fear is S6, and sadness is S7, then Fn = (a×S1) + (b×S2) + (c×S3) + (d×S4) + (e×S5) + (f×S6) + (g×S7). a to g are coefficients assigned a weight to each emotion for each activity. That is, a is the coefficient for seriousness, b for happiness, c for anger, d for disgust, e for surprise, f for fear, and g for sadness. For example, in events like concerts, the coefficient a for happiness is given a higher weight.

[0156] The data described above are examples of data generated by the data processing unit 150. The data processing unit 150 generates data based on instructions from the user input via the operation unit 105. These user instructions may be, for example, data generated by selecting from pre-prepared items.

[0157] The data processed by the data processing unit 150 (processed data) is linked to the composite image data from the processing source and recorded in the database 200 (reference). Figure 11 ).

[0158] The heatmap generation unit 160 generates a heatmap from the data processed by the data processing unit 150. The heatmap generated by the image data processing apparatus 100 of this embodiment is a graph that uses color or color intensity to represent data about the audience at each location within the venue. For example, an emotion intensity heatmap is generated by using color or color intensity to represent the emotion level of the audience at each location. Furthermore, an excitement intensity heatmap is generated by using color or color intensity to represent the excitement level of the audience at each location.

[0159] Figure 14 This is an example of a heatmap representing the level of excitement.

[0160] exist Figure 14 In this study, a heatmap is generated using a seating chart of the event venue. A seating chart is a planar representation of the seating arrangement within the event venue. The location of each seat corresponds to the location of each audience member. The location of each seat in the seating chart corresponds one-to-one with the coordinate location of each audience member in the composite image data. Therefore, a heatmap of audience level can be generated by using color or shades of color to represent the level of excitement of each audience member at their seat location.

[0161] Figure 15 This is an example of a way of displaying excitement levels. Figure 15 The image shows an example of using varying shades to represent arousal levels. Within a calculable range, arousal is divided into multiple regions. The concentration represented by each divided region is set. Figure 15 Examples are shown when the excitation level is calculated using values ​​from 1 to 100, and when it is displayed in 10 zones. Furthermore, examples are shown where the concentration increases as the excitation level increases.

[0162] The data of the heatmap generated by the heatmap generation unit 160 is linked with the data of the generation source and recorded in the database 200 (see reference). Figure 11 ).

[0163] The display control unit 170 displays the data generated by the data processing unit 150 on the display unit 106 according to the display instructions input by the user through the operation unit 105. Furthermore, it displays the heat map generated by the heat map generation unit 160 on the display unit 106.

[0164] The output control unit 180 outputs data generated by the data processing unit 150 to the external machine 300 based on the output instruction from the user input through the operation unit 105. Furthermore, it outputs data from the heatmap generated by the heatmap generation unit 160 to the external machine 300.

[0165] [effect]

[0166] Figure 16 This is a flowchart illustrating the image data processing steps in the image data processing system of this embodiment.

[0167] First, the cameras C of the audience photography device 10 photograph each area V1 to V6 within the venue (step S1). Each area V1 to V2 is photographed from multiple directions by multiple cameras.

[0168] The image data processing device 100 inputs image data captured by each camera C (step S2). The image data from each camera C will be input centrally after the activity ends. Alternatively, it can be configured for real-time input.

[0169] The image data processing device 100 processes the image data of each input camera C separately, and detects the faces of each audience member in the image represented by each image data (step S3).

[0170] The image data processing device 100 identifies the personal attributes of each audience member from the detected faces (step S4).

[0171] The image data processing apparatus 100 generates image data for each image based on the recognition results of the character attributes of each audience member in each image data (step S5). The image data is generated by recording the character attribute information of each audience member in correspondence with the position information of each audience member in the image.

[0172] The generated graph data here does not necessarily need to record the attributes of all audience members. There may be situations where faces are obscured by obstacles, and it is not necessary to be able to identify the attributes of all audience members at all times.

[0173] Therefore, after generating image data from each image data, the image data processing apparatus 100 interpolates data between image data with overlapping regions (step S6). That is, for information such as audience character attributes that are missing in one image data, interpolation is performed using information recorded in another image data. As a result, missing data generated in the image data can be suppressed.

[0174] The image data processing device 100 synthesizes the interpolated map data and generates composite map data representing the entire site (step S7).

[0175] The image data processing device 100 processes the composite image data to generate data as instructed by the user (step S8). For example, it generates data on the emotional level and excitement level of each viewer.

[0176] The image data processing apparatus 100 generates a heat map from the generated data according to the instructions from the user (step S9).

[0177] The image data processing device 100 displays the generated heat map on the display unit 106 or outputs it to the external machine 300 according to the instructions from the user (step S10).

[0178] As explained above, the image data processing system 1 according to this embodiment generates image data that includes information on the attributes of all spectators within the venue. Therefore, even when generating image data for a large venue, accurate image data can be generated efficiently. Furthermore, compared to generating image data for all spectators at once, the processing burden is reduced.

[0179] Furthermore, since each graph dataset contains at least some overlapping information about the audience's character attributes, information missing in one graph dataset can be interpolated using another graph dataset. Thus, information about each audience member's character attributes can be collected comprehensively from each graph dataset.

[0180] [Variation Example]

[0181] (1) Shooting method

[0182] In the above embodiment, the viewing area of ​​the venue is divided into multiple zones, and multiple cameras capture the composition of each zone from multiple directions. However, the method of photographing the audience within the venue is not limited to this. It is sufficient that at least two cameras are used to photograph the structure of each audience member. This allows for interpolation.

[0183] Figure 17 This is another example of a method for photographing the audience.

[0184] exist Figure 17 In the diagram, boxes W1 to W3 represent the camera-based shooting range. For example... Figure 17 As shown, in this example, within area Vc, the shooting range of each camera is set to overlap at least partially. Furthermore, within area Vc, the shooting range of each camera is set to allow at least two cameras to capture each viewer.

[0185] Each camera preferably captures overlapping areas under different conditions. For example, as described in the above embodiment, it is configured to capture overlapping areas from mutually different directions. Thus, in an image captured by one camera, even if the viewer's face is obscured by an obstacle, it can still be captured by another camera.

[0186] Furthermore, it is possible to configure the image to capture repeated areas with different exposures. In this case, it is also possible to configure the image to be captured from approximately the same direction. By capturing repeated areas with different exposures, for example, even if a face cannot be detected in an image captured by one camera due to flash and / or ghosting (sunlight, reflection, flash, etc.), it is possible to detect it in an image captured by another camera.

[0187] Regarding exposure, in addition to adjusting by changing the aperture value, shutter speed, or ISO, it can also be adjusted by using filters such as ND filters (Neutral Density Filters).

[0188] (2) Taking pictures

[0189] In the above embodiments, the example described is for capturing moving images and processing them in frame units, but the present invention can also be applied to capturing and processing still images.

[0190] Furthermore, the dynamic images also include cases where still images are continuously captured and processed within a preset time interval. For example, it also includes cases where processing is performed by interval shooting, time-lapse dynamic image shooting, etc.

[0191] (3) Character attributes

[0192] In the above embodiments, the identification of age, gender, and emotion of each viewer was used as an example of facial recognition attributes, but facial recognition attributes are not limited to this. Furthermore, for example, it can include personal identification information of individual viewers. That is, it can include information that identifies an individual. Regarding the identification of personal identification information, for example, a facial recognition database that associates facial images with personal identification information can be used. Specifically, this is done by performing a verification process between the detected facial image and facial images stored in the facial recognition database, and obtaining the personal identification information corresponding to the matching facial image from the facial recognition database. The personal identification information can associate information such as the viewer's age and gender. Therefore, it is not necessary to identify age and gender when identifying personal identification information.

[0193] (4) Interpolation processing of graph data

[0194] Interpolation of graphical data is performed between graphical data that share information about the same person's attributes. This type of graphical data is generated from image data with overlapping photographic ranges.

[0195] Regarding graph data interpolation, basically, for character attribute information missing in one graph dataset, interpolation is performed using another graph dataset. Furthermore, even when all attributes are present, interpolation of character attribute information can be performed in the following manner.

[0196] (a) Using information on person attributes with high recognition accuracy

[0197] When the attribute information of the same person exists in multiple graph datasets, the attribute information with relatively high recognition accuracy is used to replace the attribute information with relatively low recognition accuracy when interpolating the attribute information of each person. More specifically, the attribute information with the highest recognition accuracy is used. In this case, the information of all graph datasets except those with the attribute information with the highest recognition accuracy is rewritten.

[0198] In this case, the person attribute recognition unit 120C calculates its recognition accuracy along with the recognition of person attributes. The algorithm for calculating the recognition accuracy (reliability, also known as the evaluation value, etc.) can employ algorithms well-known in image recognition.

[0199] (b) Take the average of the attributes of the same person across the data in each graph.

[0200] The character attribute of a person is determined by averaging the character attributes of the same person across graph data containing information about their character attributes. In this case, the calculated average is used to replace the information in each graph data set.

[0201] (c) Employ a weighted average that corresponds to the recognition accuracy.

[0202] When calculating the average of the attributes of the same person, a weight corresponding to the recognition accuracy of the attribute is added, and the average is calculated. The higher the recognition accuracy of the attribute, the greater the weight is assigned.

[0203] The methods described above can also be used when interpolating character attribute information for characters missing in one graph dataset from other graph datasets. That is, they can be used when there are multiple graph datasets containing character attribute information for characters missing in one graph dataset.

[0204] Furthermore, using person attributes with low recognition accuracy could reduce the reliability of the graph data. Therefore, when performing interpolation, it is preferable to use only information on person attributes with recognition accuracy above a certain threshold. This threshold is preferably set by the user. This threshold is an example of a first threshold.

[0205] (5) Selection of interpolation method

[0206] The method used to interpolate the graph data can be set to be freely selectable by the user. In this case, for example, the available interpolation methods can be displayed on the display unit 106 so that the user can select them via the operation unit 105.

[0207] Furthermore, the system can be configured to generate a heatmap whenever the interpolation method is switched, and display the generated heatmap on the display unit. This allows for easy selection of the preferred interpolation method.

[0208] Furthermore, the image data processing device 100 can automatically determine the optimal interpolation method and employ the selected method. The following methods can be considered as a method for automatically determining the interpolation method.

[0209] (a) Select the number of character attributes extracted from the specified time and / or specified area of ​​the character, which is the largest interpolation method.

[0210] (b) The average of the recognition accuracy of the person's attributes extracted from the person in the specified time and / or specified area, including the whole, is selected as the interpolation method with the highest accuracy.

[0211] (c) Select the interpolation method that minimizes the variation in the recognition accuracy of the person's attributes extracted from the person within a specified time and / or a specified area, including the whole.

[0212] (d) Select an interpolation method that can extract all the character attributes of the specified character.

[0213] (e) The average of the recognition accuracy of the character attributes of the specified character is selected as the interpolation method with the highest accuracy.

[0214] (f) Select the interpolation method that minimizes the variation in the recognition accuracy of the character attributes of the specified character.

[0215] The user can specify the time, area, and personnel through the operation unit 105.

[0216] (6) Handling situations where interpolation cannot be performed from any graph data

[0217] It is possible that no camera can capture the audience. In this case, information about the person's attributes is missing for that time period. When a person's attributes are missing for a given time period, the following method is used to interpolate the information about those attributes.

[0218] First, calculate the temporal changes in the character attributes of all viewers. Next, identify the viewers whose character attribute information is missing within a given time period. Then, identify viewers whose character attribute changes are similar to those of the missing viewers. Finally, use the character attribute information of the identified viewers to interpolate the character attribute information for the missing time periods.

[0219] This method is effective in identifying emotions as character attributes. That is, it interpolates missing emotional information using information from characters exhibiting similar emotional changes. This is assumed to be because they show similar responses to emotions.

[0220] (7) Correction of interpolated graph data

[0221] The interpolated graph data can be further refined for use. For example, individuals with low accuracy in identifying their attributes throughout the entire event can be excluded from the graph data. This improves the reliability of the interpolated graph data. This process can be implemented, for example, as follows.

[0222] First, the recognition accuracy of all characters' attributes is calculated over the entire duration of the event. Next, characters whose recognition accuracy falls below a predetermined value are identified as having recognition accuracy above that value. The character attribute information of these identified characters is then excluded from the interpolated graph data. The predetermined value is an example of the second threshold.

[0223] (8) Methods for identifying people who are repeated in the graph data

[0224] When interpolating graph data, it is necessary to identify duplicate characters across graph data. In this case, the positional information of each character recorded in the graph data is used to determine the duplicate characters. That is, the configuration relationship (coordination pattern) of each character can be determined from the positional information of each character recorded in each graph data, and therefore duplicate characters can be identified based on the configuration relationship. Similarly, duplicate characters can also be identified from the character attribute information at each position. That is, duplicate characters can be identified from the pattern of character attributes.

[0225] Furthermore, when generating composite image data, it is also possible to perform composite processing on individual image data by identifying characters that appear repeatedly in different image data sets. That is, composite processing can be performed without using information such as camera configuration and position.

[0226] (9) Heatmap

[0227] In the above implementation, although a seating chart of the event venue is used to generate a heat map, the method of generating the heat map is not limited to this. It is acceptable to use color or shades of color to represent the audience data at each location generated from the composite image data.

[0228] Furthermore, as a heatmap display method, it doesn't need to show the entire image; it can also be set to display each region separately. Additionally, the heatmap can be overlaid with the actual video.

[0229] (10) Composition of image data processing device

[0230] In image data processing devices, the hardware structure of the processing units that perform various processes is implemented by various processors. These processors include general-purpose processors that execute programs to function as various processing units, such as CPUs and / or GPUs (Graphics Processing Units); FPGAs (Field-Programmable Gate Arrays), which can have their circuit structure modified after manufacturing (Programmable Logic Devices, PLDs); and ASICs (Application Specific Integrated Circuits), which have circuit structures specifically designed to perform specific processes (dedicated circuits). The terms "program" and "software" have the same meaning.

[0231] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types. For example, a processing unit can be composed of multiple FPGAs or a combination of a CPU and an FPGA. Furthermore, multiple processing units can be composed of a single processor. Examples of multiple processing units composed of a single processor include: First, as exemplified by computers such as client computers or servers, a processor is composed of a combination of one or more CPUs and software, which functions as multiple processing units. Second, as exemplified by systems-on-a-chip (SoCs), a processor that implements the overall system functionality including multiple processing units is used on a single IC (Integrated Circuit) chip. Thus, various processing units are configured as hardware structures using one or more of the aforementioned processors.

[0232] Symbol Explanation

[0233] 1-Image data processing system, 2-Event venue, 3-Performer, 4-Stage, 6-Seats, 10-Audience photography device, 100-Image data processing device, 101-CPU, 103-ROM, 104-HDD, 105-Operating unit, 106-Display unit, 107-Input / output interface, 110-Photography control unit, 120-Image data processing unit, 120A-Photography information acquisition unit, 120B-Face detection unit, 1 20C - Person Attribute Recognition Department, 120D - Distribution Generation Department, 130 - Graph Data Interpolation Department, 140 - Graph Data Synthesis Department, 150 - Data Processing Department, 160 - Heatmap Generation Department, 170 - Display Control Department, 180 - Output Control Department, 200 - Database, 300 - External Machine, C - Camera, C1 - First Camera, C2 - Second Camera, C3 - Third Camera, C4 - Fourth Camera, C5 - Fifth Camera, C6 - Sixth Camera, F - The bounding box surrounding the detected face, Im - Image, P - Viewer, P29 - Viewer, P34 - Viewer, P47 - Viewer, P55 - Viewer, P62 - Viewer, P84 - Viewer, R1 - Camera range of the first camera, R2 - Camera range of the second camera, R3 - Camera range of the third camera, R4 - Camera range of the fourth camera, R5 - Camera range of the fifth camera, R6 - Camera range of the sixth camera, V - Viewing area, V1 - Area divided into viewing areas, V2 - Area divided into viewing areas, V3 - Area divided into viewing areas, V4 - Area divided into viewing areas, V5 - Area divided into viewing areas, V6 - Area divided into viewing areas, Vc - Area divided into viewing areas, W1 - Box representing the camera range, W2 - Box representing the camera range, W3 - Box representing the camera range, S1~S10 - Image data processing steps in the image data processing system.

Claims

1. An image data processing apparatus for processing at least a portion of repetitive image data within a photographic range acquired by a plurality of photographic devices, the image data processing apparatus comprising: processor, The processor performs the following processing: The process of detecting the face of a person in the image represented by each image data and identifying the person's attributes based on the detected face; The process of generating image data for each image data, which records the identified person attributes in relation to the position of the person within the image represented by the image data; Interpolation processing is performed on the character attributes of the character that are repeated among multiple graph data; and The process of generating composite graph data by combining multiple interpolated graph data. If it is impossible to interpolate the information of the character attributes using other graph data, the processor uses information of other character attributes whose information changes over time in a similar manner.

2. The image data processing apparatus according to claim 1, wherein, The processor further performs the process of generating a heatmap from the composite graph data.

3. The image data processing apparatus according to claim 2, wherein, The processor further performs the process of displaying the generated heatmap on a display.

4. The image data processing apparatus according to claim 2 or 3, wherein, The processor further performs the process of outputting the generated heatmap to the outside.

5. The image data processing apparatus according to any one of claims 1 to 3, wherein, The processor checks the character attributes of the characters that are repeated among the multiple graph data, and for the character attributes of the characters that are missing in one graph data, it interpolates them using the character attributes of the characters in another graph data.

6. The image data processing apparatus according to any one of claims 1 to 3, wherein, The processor calculates the recognition accuracy while recognizing the attributes of the person.

7. The image data processing apparatus according to claim 6, wherein, The processor interpolates the character attributes of duplicate characters by replacing the character attributes of characters with relatively low recognition accuracy with the character attributes of characters with relatively high recognition accuracy.

8. The image data processing apparatus according to claim 6, wherein, The processor calculates the average of the attributes of each person by assigning weights corresponding to the recognition accuracy, and uses the calculated average to interpolate the attributes of repeated persons.

9. The image data processing apparatus according to claim 6, wherein, The processor has multiple recognition accuracies, and uses information about the attributes of the person with a recognition accuracies of at least a first threshold to interpolate the attributes of repeated persons.

10. The image data processing apparatus according to claim 6, wherein, The processor further performs processing on the interpolated graph data to exclude information on the person attributes with recognition accuracy below a second threshold.

11. The image data processing apparatus according to any one of claims 1 to 3, wherein, The processor further performs a process of identifying the characters that are repeated among the multiple graph data.

12. The image data processing apparatus according to claim 11, wherein, The processor determines which characters are repeated across multiple sets of graph data based on the configuration relationships of the characters in the graph data.

13. The image data processing apparatus according to claim 11, wherein, The processor determines the repeated characters across multiple sets of graph data based on the character attributes of the characters at each location in the graph data.

14. The image data processing apparatus according to any one of claims 1 to 3, wherein, The processor identifies at least one of gender, age, and emotion based on the person's face as attributes of the person.

15. The image data processing apparatus according to any one of claims 1 to 3, wherein, The processor instructs multiple photographic devices to photograph the overlapping areas of the photographic range under mutually different conditions.

16. The image data processing apparatus according to claim 15, wherein, The processor instructs multiple photographic devices to photograph overlapping areas of the photographic range from mutually different directions.

17. The image data processing apparatus according to claim 15, wherein, The processor instructs multiple photographic devices to photograph the overlapping areas of the photographic range with different exposures.

18. An image data processing system, comprising: Multiple photographic devices that repeat at least a portion of the photographic range; and An image data processing apparatus that processes image data acquired by the plurality of said photographic devices, wherein, The image data processing device includes a processor. The processor performs the following processing: The process of detecting the face of a person in the image represented by each image data and identifying the person's attributes based on the detected face; The process of generating image data for each image data, which records the identified person attributes in relation to the position of the person within the image represented by the image data; Interpolation processing is performed on the character attributes of the character that are repeated among multiple graph data; and The process of generating composite graph data by combining multiple interpolated graph data. If it is impossible to interpolate the information of the character attributes using other graph data, the processor uses information of other character attributes whose information changes over time in a similar manner.

19. The image data processing system according to claim 18, wherein, Multiple photographic devices photograph the overlapping areas of the photographic range under different conditions.

20. The image data processing system according to claim 19, wherein, Multiple photographic devices capture images of overlapping areas from different directions.

21. The image data processing system according to claim 19 or 20, wherein, Multiple photographic devices photograph the overlapping areas of the photographic range with different exposures.

Images