Information Processing System and Method

Abstract

The present invention provides an information processing system and method capable of easily grasping the situation of a person within a specified area. The information processing system includes: a first photographing unit that photographs a performer; a second photographing unit that photographs a person within a specified area; a first recognition unit that recognizes a scene based on an image photographed by the first photographing unit; a second recognition unit that recognizes the expression of a person based on an image photographed by the second photographing unit; a calculation unit that calculates a situation index of a person corresponding to the scene based on the recognition result of the scene and the recognition result of the expression of the person; a heat map creation unit that creates a heat map representing the correspondence between the situation index of the person and the position of the person within the area; and an output unit that outputs the heat map.

Description

Technical Field

[0001] The present invention relates to an information processing system and method. Background Art

[0002] In Patent Document 1, a technique is described in which, in a service for live-transmitting video such as a concert, a heat map visualizes the excitement level of users watching the video and presents it to actors at the venue or the like.

[0003] Patent Document 2 describes a technique for calculating the excitement level of a meeting based on meeting data recorded as sound and / or video.

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: International Publication No. 2016 / 009865

[0007] Patent Document 2: Japanese Patent Application Laid-Open No. 2016-12216 Summary of the Invention

[0008] One embodiment of the technology of the present invention provides an information processing system and method capable of easily grasping the situation of people within a specified area.

[0009] Means for Solving the Technical Problem

[0010] (1) An information processing system includes: a first photographing unit that photographs a performer; a second photographing unit that photographs people within a specified area; a first recognition unit that recognizes a scene based on an image photographed by the first photographing unit; a second recognition unit that recognizes the expressions of people based on an image photographed by the second photographing unit; a calculation unit that calculates a situation index of people corresponding to the scene based on the recognition result of the scene and the recognition result of the expressions of people; a heat map creation unit that creates a heat map showing the correspondence between the situation index of people and the positions of people within the area; and an output unit that outputs the heat map.

[0011] (2) The information processing system according to (1) further includes: a setting unit that sets the number of clusters according to the influence power of the performer; and a clustering unit that clusters the heat map according to the set number of clusters, and the output unit outputs the clustered heat map.

[0012] (3) In the information processing system according to (2), the clustering unit clusters the heat map using data of people whose situation index is equal to or greater than a threshold value.

[0013] (4) The information processing system according to (2) or (3) further includes an estimation unit that estimates the influence power of the performer based on an image photographed by the first photographing unit.

[0014] (5) The information processing system according to any one of (2) to (4), wherein the spreading power includes at least one of the skills and spare capacity possessed by the performer.

[0015] (6) The information processing system according to any one of (1) to (5), further comprising a conversion processing unit that converts the image of the heat map into an image with a viewpoint different from that of the second photographing unit.

[0016] (7) The information processing system according to any one of (1) to (6), further comprising an inversion processing unit that inverts the color or shading of the heat map.

[0017] (8) The information processing system according to any one of (1) to (7), further comprising a display unit that displays the heat map output from the output unit.

[0018] (9) The information processing system according to any one of (1) to (7), further comprising a projection unit that projects the heat map output from the output unit onto an area.

[0019] (10) The information processing system according to (9), wherein the projection unit projects the heat map onto the area by projection mapping.

[0020] (11) An information processing method, comprising: a step of photographing a performer; a step of photographing a person within a prescribed area; a step of identifying a scene based on the image of the photographed performer; a step of identifying the expression of the person based on the image of the person within the photographed area; a step of calculating a condition index of the person corresponding to the scene based on the identification result of the scene and the identification result of the expression of the person; a step of creating a heat map representing the correspondence between the condition index of the person and the position of the person within the area; and a step of outputting the heat map.

[0021] (12) The information processing method according to (11), further comprising: a step of setting the number of groups based on the information of the spreading power possessed by the performer; and a step of clustering the heat map according to the set number of groups, wherein in the step of outputting the heat map, the clustered heat map is output. Description of the Drawings

[0022] Figure 1 is a diagram showing the schematic structure of the information processing system.

[0023] Figure 2 is a block diagram showing an example of the hardware structure of the information processing device.

[0024] Figure 3 is a block diagram of the functions implemented by the information processing device.

[0025] Figure 4 It is a conceptual diagram of face detection based on the face detection unit.

[0026] Figure 5 It is a conceptual diagram of the recognition of facial expressions based on the expression recognition unit.

[0027] Figure 6 It is a diagram showing an example of the result of expression recognition.

[0028] Figure 7 It is a diagram showing the relationship between the degree of excitement and the display on the heat map.

[0029] Figure 8 It is a diagram showing an example of the image captured by the audience photography device.

[0030] Figure 9 It is a diagram showing an example of the generated heat map.

[0031] Figure 10 It is a diagram showing the positional relationship between the heat map and the audience.

[0032] Figure 11 It is a flowchart showing the steps of information processing.

[0033] Figure 12 It is a block diagram of the functions implemented by the information processing device.

[0034] Figure 13 It is a conceptual diagram of cluster processing.

[0035] Figure 14 It is a conceptual diagram of cluster processing.

[0036] Figure 15 It is a conceptual diagram of cluster processing.

[0037] Figure 16 It is a flowchart showing the steps of information processing.

[0038] Figure 17 It is a conceptual diagram of the change of the heat map based on cluster processing.

[0039] Figure 18 It is a block diagram of the functions implemented by the information processing device.

[0040] Figure 19 It is a block diagram of the functions implemented by the information processing device.

[0041] Figure 20 It is a block diagram of the functions implemented by the information processing device.

[0042] Figure 21 It is a diagram showing an example of the conversion of the heat map image.

[0043] Figure 22It is a block diagram of the functions implemented by the information processing device.

[0044] Figure 23 It is a diagram showing an example of the inversion process of the image representing the heat map.

[0045] Figure 24 It is a diagram showing the schematic structure of the information processing system. Detailed implementation mode

[0046] Hereinafter, the preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0047] [First Embodiment]

[0048] A performer who performs in front of a large number of audiences controls the performance content while confirming the situation of the audiences. The ability of the performer to grasp the situation of the audiences largely depends on experience. And even an experienced performer may have difficulty accurately grasping the whole if the number of audiences increases excessively. In this embodiment, an information processing system and method that can easily grasp the situation of the audiences are provided.

[0049] [System Structure]

[0050] Figure 1 It is a diagram showing the schematic structure of the information processing system of this embodiment.

[0051] As shown in this diagram, the information processing system 10 of this embodiment is configured to include a performer photographing device 100, an audience photographing device 200, an information processing device 300, a display device 400, etc.

[0052] [Performer Photographing Device]

[0053] The performer photographing device 100 photographs the performer 1. The performer 1 refers to the person who performs. In addition, the performance here is not limited to artistic performance behaviors such as singing, acting, and playing, but also widely includes behaviors such as speeches and lectures. That is, it means performing a certain performance behavior for the audiences.

[0054] The performer photographing device 100 is an example of the first photographing unit. The performer photographing device 100 is composed of at least one camera. The camera is composed of a so-called video camera (including a digital camera with a video photographing function (the function of photographing sequential images)), and continuously photographs the performer at a predetermined frame rate.

[0055] The images captured by the performer photography device 100 are used to identify the scene. That is, to identify what scene the performance by the performer 1 is. Therefore, the performer photography device 100 is configured and set in a manner suitable for its use. The necessary conditions for the images required for scene recognition vary depending on the content, scale, etc. of the performance. Therefore, the camera that constitutes the performer photography device 100 is appropriately selected according to the content, scale, etc. of the performance and set at the optimal position.

[0056] [Audience photography device]

[0057] The audience photography device 200 photographs the audience 2. The audience photography device 200 photographs the audience area 3 where the audience 2 is located from a certain position and photographs the audience. The audience 2 is a person who watches (views, watches, listens to, etc.) the performance of the performer 1. The audience area 3 where the audience 2 is located is an example of a specified area. The audience 2 is an example of a person within the specified area.

[0058] The audience photography device 200 is an example of the second photography unit. The audience photography device 200 is composed of at least one camera. In the case where the audience area 3 cannot be photographed by one camera, the audience photography device 200 is composed of multiple cameras. In this case, the audience area 3 is divided into multiple areas, and multiple cameras share the photographing of each area. In addition, it can also be configured such that multiple cameras photograph the same area. For example, it can be configured such that multiple cameras photograph the same area of the audience 2 from different directions. The camera is composed of a so-called imaging device and continuously photographs the audience 2 within the area that is the object. The photography is performed at the same frame rate as the performer photography device 100 and is synchronized. The synchronization at this time does not mean complete synchronization in frame units.

[0059] The images captured by the audience photography device 200 are used to identify the expressions of the audience. Therefore, the camera that constitutes the audience photography device 200 is set at a position where it can photograph the faces of the audience 2 within the area that is the photography object and has the performance of being able to identify the expressions of the audience based on the captured images. That is, it has sufficient resolution performance required to identify the expressions of the audience based on the captured images.

[0060] [Information processing device]

[0061] The information processing device 300 inputs the images of the performer captured by the performer photography device 100 and the images of the audience captured by the audience photography device 200, creates a heat map representing the excitement state of the audience 2, and outputs it to the display device 400.

[0062] Figure 2 It is a block diagram showing an example of the hardware structure of the information processing device.

[0063] The information processing device 300 is composed of a computer including a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, an HDD (Hard Disk Drive) 304, an operation unit (such as a keyboard, a mouse, a touch panel, etc.) 305, a display unit (such as a liquid crystal display, etc.) 306, an input interface (interface, I / F) 307, an output interface 308, and the like. The image data of the performer captured by the performer photographing device 100 and the image data of the audience captured by the audience photographing device 200 are input into the information processing device 300 via the input interface 307. The heat map produced by the information processing device 300 is output to the display device 400 via the output interface 308.

[0064] Figure 3 It is a block diagram of the functions implemented by the information processing device.

[0065] As shown in this figure, the information processing device 300 has the functions of a first image input unit 311, a second image input unit 312, a scene recognition unit 313, a face detection unit 314, an expression recognition unit 315, an excitement level calculation unit 316, a heat map production unit 317, and a heat map output unit 318. Each function is implemented by the CPU as a processor executing a prescribed program.

[0066] The first image input unit 311 receives the input of the image captured by the performer photographing device 100. The image captured by the performer photographing device 100 is an image of the performer 1. The image captured by the performer photographing device 100 is input into the information processing device 300 via the input interface 307.

[0067] The second image input unit 312 receives the input of the image captured by the audience photographing device 200. The image captured by the audience photographing device 200 is an image of the audience area 3 and is an image of the audience 2. The image captured by the audience photographing device 200 is input into the information processing device 300 via the input interface 307. The input of the image is performed synchronously with the first image input unit 311.

[0068] The scene recognition unit 313 is an example of the first recognition unit. The scene recognition unit 313 recognizes the scene of the performance by the performer 1 based on the image of the performer 1 captured. The scene recognition unit 313 recognizes the scene within a pre-determined range of categories. For example, scenes that can be recognized such as a scene that makes the audience laugh, a scene that makes the audience serious, a scene that makes the audience excited, and a scene that makes the audience angry are pre-determined, and the scene is recognized within this determined range. The recognition of the scene can adopt known techniques. For example, a method of using an image recognition model generated by machine learning, deep learning, etc. to recognize the scene can be adopted. The scene is recognized at a pre-determined time interval. For example, it is the time interval of the frames of the input image.

[0069] The face detection unit 314 detects the face of the audience 2 from the image of the audience 2 captured. Figure 4 It is a conceptual diagram based on the face detection by the face detection unit. The face detection unit 314 detects the face of each audience 2 from the image I2 of the audience 2 captured, and determines the position of each detected face. Face detection can adopt known techniques. The position of the face is determined by the coordinate position (x, y) within the image. Specifically, in the case of the frame F that encloses the detected face, it is determined by the coordinate position (x F , y F ). The coordinate position within the image corresponds to the actual position of the audience. Face detection is performed, for example, by scanning sequentially from the upper left to the lower right of the image. And the detected faces are numbered in the order of detection. Face detection is performed at a pre-determined time interval. For example, it is the time interval of the frames of the input image.

[0070] The expression recognition unit 315 is an example of the second recognition unit. Figure 5 It is a conceptual diagram based on the recognition of the facial expression by the expression recognition unit. The expression recognition unit 315 recognizes the facial expression of the audience 2 based on the image IF of the face of the audience 2 detected by the face detection unit 314. The recognition of the expression means discriminating the type of the expression. The type of the expression is represented by words indicating emotions. Therefore, the recognition of the expression means determining the type of the expression by words indicating emotions. The determination of the expression can be a determination based on a word indicating a single emotion, or a determination based on a combination of words indicating emotions. In the case of combining words indicating emotions, weights can also be assigned to the words indicating each emotion. In the present embodiment, the facial expression is classified into three types: "laugh", "anger", and "sadness". As the expression recognition result, a score (expression score) that numerically represents the degree (also referred to as expression similarity) of each expression is output. For example, the expression score is output with a maximum value of 100. In addition, the expression score can be output in such a way that the sum of the degrees of each expression becomes 100.

[0071] Recognition of expressions can adopt well-known techniques. For example, similar to the recognition of scenes, a method of using an image recognition model generated by machine learning, deep learning, etc. to recognize expressions can be adopted.

[0072] Figure 6 It is a diagram showing an example of the result of expression recognition. As shown in this diagram, by performing expression recognition by the expression recognition unit 315, an expression score is obtained for each viewer. In addition, the position of each viewer is determined by the face detection unit 314 based on the coordinate position within the image.

[0073] The excitement level calculation unit 316 calculates the excitement level of each viewer corresponding to the scene based on the recognition results of the scene recognition unit 313 and the expression recognition unit 315. The excitement level calculation unit 316 is an example of a calculation unit. The excitement level numerically represents the excitement level (degree of excitement) of each viewer. The excitement level is an example of a situation index. The excitement level is calculated based on the expression score using a pre-determined arithmetic expression. For example, when the expression score for smiling is set as S1, the expression score for anger is set as S2, and the expression score for sadness is set as S3, the arithmetic expression Fn is defined as Fn = a × S1 + b × S3 + c × S4. a, b, and c are coefficients (weights) determined for each scene. a is the coefficient for the smiling expression, b is the coefficient for the angry expression, and c is the coefficient for the sad expression. For example, assume that the coefficients a, b, and c for smiling, anger, and sadness determined for a certain scene are a = 0.9, b = 0.05, and c = 0.05. And assume that the expression scores of a certain viewer are smiling: 100, anger: 20, and sadness: 10. In this case, using the above arithmetic expression, the excitement level of this viewer in this scene becomes Fn = 0.9 × S1 + 0.05 × S3 + 0.05 × S4 = 0.9 × 100 + 0.05 × 20 + 0.05 × 10 = 91.5. And assume that the expression scores of a certain viewer are smiling: 30, anger: 20, and sadness: 20. In this case, using the above arithmetic expression, the excitement level of this viewer in this scene becomes Fn = 0.9 × 30 + 0.05 × 20 + 0.05 × 20 = 29. Information on the coefficients a, b, and c is stored in the ROM 302, RAM 303, or HDD 304 for each scene.

[0074] The heat map creation unit 317 creates a heat map based on the information on the excitement level of each viewer calculated by the excitement level calculation unit 316. The heat map is created by establishing a corresponding association between the excitement level of each viewer and the position of each viewer. The excitement level is represented by color or shading.

[0075] Figure 7It is a diagram showing the relationship between the degree of excitement and the display on the heat map. This diagram shows an example when the degree of excitement is represented by shades. Within the computable range, the degree of excitement is divided into multiple categories. For each divided category, the display concentration is determined. This diagram shows an example when the degree of excitement is calculated using values from 1 to 100, and also shows an example when it is divided into 10 categories for display. Moreover, it shows an example when the display concentration increases as the degree of excitement becomes higher.

[0076] Here, an example of the heat map produced by the heat map production unit 317 will be described. Here, consider a situation where a certain performer performs in front of a large number of audiences at a certain event venue. For example, it is a situation where a singer sings at a concert venue.

[0077] Figure 8 It is a diagram showing an example of the image captured by the audience photography device.

[0078] In this diagram, symbol 2 represents the audience, and symbol 3 represents the audience area. Audience 2 watches the performance by the performer on the seats arranged in the audience area 3.

[0079] Figure 9 It is a diagram showing an example of the produced heat map. And Figure 10 It is a diagram showing the positional relationship between the heat map and the audience.

[0080] As shown in this diagram, the heat map HM is produced by establishing a corresponding association between the degree of excitement of each audience 2 and the position of each audience 2 by using colors or the shades of specific colors. By representing the degree of excitement of each audience 2 with the heat map HM, the degree of excitement of the entire venue can be visually presented. Thereby, the state of the audience 2 can be easily grasped.

[0081] The heat map output unit 318 is an example of the output unit. The heat map output unit 318 outputs the heat map HM produced by the heat map production unit 317 to the display device 400.

[0082] [Display device]

[0083] The display device 400 is an example of the display unit. The display device 400 displays the heat map output from the information processing device 300. The display device 400 is constituted by, for example, a flat panel display such as a liquid crystal display (Liquid Crystal Display), a plasma display, an organic EL display (Organic ElectroLuminescence display, Organic Light Emitting Diode display), a field emission display (Field Emission Display), an electronic paper, or a projector and a screen (or an equivalent of the screen). The display device 400 is arranged at a position where the performer 1 can visually recognize it.

[0084] [Function]

[0085] Figure 11 It is a flowchart showing the steps (information processing method) of information processing of the information processing system based on this embodiment.

[0086] First, the performer photographing device 100 photographs the performer 1 (step S11). In parallel with this, the audience photographing device 200 photographs the audience area 3 and photographs the audience 2 within the audience area 3 (step S13).

[0087] Next, the scene of the performance by the performer 1 is recognized based on the image of the performer 1 photographed by the performer photographing device 100 (step S12). And, the faces of the respective audiences 2 are detected from the image of the audience area 3 photographed by the audience photographing device 200 (step S14), and the expressions of the detected faces are recognized (step S15).

[0088] Next, the excitement level of each audience 2 corresponding to the scene is calculated based on the recognition result of the scene and the recognition result of the expression (step S16). That is, using the conversion formula corresponding to the scene and the expression, the expression scores of each audience 2 are converted into the excitement level, and the excitement level of each audience 2 is obtained.

[0089] Next, a heat map is created based on the information of the excitement level of each audience 2 obtained (step S17). The heat map is created by establishing a corresponding association between the excitement level possessed by each audience 2 and the position of each audience 2 (refer to Figure 9 ).

[0090] Next, the created heat map is output to the display device 400 and displayed on the screen (step S18). The performer 1 can grasp the excitement state of the audience in the audience area 3 by visually recognizing the heat map displayed on the display device 400. Thereby, it is possible to easily determine the content of the performance to be carried out. That is, for example, when there is uneven excitement, the performance is carried out so that there is no unevenness in the entire venue. And, when the excitement is low, the performance content is changed to increase the excitement. In this way, it is possible to easily control the content of the performance based on the heat map.

[0091] [Second Embodiment]

[0092] In the information processing system of this embodiment, a heat map of the cluster is output. Except for outputting the heat map of the cluster, it is the same as the information processing system of the first embodiment above. Therefore, hereinafter, only the cluster will be described.

[0093] Figure 12 It is a block diagram of the functions realized by the information processing device of this embodiment.

[0094] As shown in the figure, the information processing apparatus 300 of the present embodiment further includes a dissemination power estimation unit 321, a group number setting unit 322, and a clustering unit 323.

[0095] The dissemination power estimation unit 321 estimates the dissemination power of the performer based on the image of the performer captured by the performer photographing apparatus 100 (first photographing unit). In this specification, "dissemination power" refers to the ability related to excitement. The dissemination power is defined by the skills and spare capacity of the performer, etc. In the present embodiment, the dissemination power is defined by the spare capacity of the performer, and the spare capacity is estimated based on the image of the performer, thereby estimating the dissemination power. The spare capacity is estimated based on the fatigue degree of the performer. Specifically, the expression of the performer is recognized, the fatigue degree is determined based on the expression, and the spare capacity is estimated. As the technology for recognizing the expression from the image and the technology for determining the fatigue degree based on the expression, well-known technologies can be adopted. For example, an image recognition model generated by machine learning, deep learning, etc. can be used for the recognition of the expression and the determination of the fatigue degree. The fatigue degree is represented by a numerical value, for example, and the spare capacity (dissemination power) is obtained based on the numerical value representing the fatigue degree. For example, using a prescribed conversion formula, the fatigue degree is converted into the spare capacity (dissemination power). In this case, the higher the fatigue degree, the lower the value to which the spare capacity is converted, and the lower the fatigue degree, the higher the value to which the spare capacity is converted. The estimation process of the dissemination power can be performed at a predetermined time interval.

[0096] The group number setting unit 322 sets the number of groups at the time of clustering based on the information of the dissemination power estimated by the dissemination power estimation unit 321. The relationship between the set number of groups and the dissemination power is defined in the form of a table or the like, for example, and stored in the ROM 302, the RAM 303, or the HDD 304. The group number setting unit 322 refers to a table or the like and determines (sets) the number of groups based on the dissemination power estimated by the dissemination power estimation unit 321.

[0097] The number of groups is set to be larger as the dissemination power is higher. That is, it is set that the higher the dissemination power indicating the spare capacity related to excitement, the larger the number of groups. This means that the more spare capacity there is, the more ability to control the performance. That is, it means that even in a situation where there is a deviation in excitement, there is the ability to make it uniform. And it means that even in a situation where the excitement is insufficient, there is the ability to make it exciting.

[0098] The clustering unit 323 clusters the heat map according to the number of groups set by the group number setting unit 322. Specifically, the data of the heat map is clustered with the set number of groups, and a heat map (clustered heat map) is created that is displayed with colors or shades differentiated for each group. The data of the heat map is data that associates the excitement level of each audience with the position of each audience. Clustering can adopt a well-known method. For example, clustering processing based on the k-means method (k-average method) can be adopted.

[0099] Figures 13 to 15 It is a conceptual diagram of the clustering process.

[0100] Figure 13 This is a diagram showing an example of data for a heat map. In this diagram, each circle represents the position of each viewer within the area. The position of each viewer (the position of the circle) is determined by the coordinate position (x, y). Also, the value within each circle indicates the level of excitement of the viewers within that circle.

[0101] First, data of viewers with an excitement level less than the threshold is omitted from the heat map data. When the threshold for the excitement level is set to 100, data of viewers with an excitement level less than 100 is omitted. In Figure 14 this, the data of the omitted viewers is represented by hollow circles, and the data of the other viewers is represented by circles painted gray.

[0102] Next, using the data of viewers with an excitement level above the threshold as the object, the heat map data is clustered (clustering process) into the set number of clusters. Specifically, the distance in the (x, y) coordinates is used as the distance for the k-means method for clustering. In this case, the distance in the (x, y) coordinates is defined by {(x_i - x) 2 + (y_i - y) 2} 0.5 defined.

[0103] Next, the average of the excitement level for each cluster is calculated. Based on the calculated average of the excitement level, the color or shading is differentiated for each cluster and displayed. Thus, as Figure 15 shown, a clustered heat map (cluster heat map) is created.

[0104] The clustering process can also use the following method. The distance for k-means is defined by the weighted sum value of the (x, y) coordinates and the excitement level, and clustering is performed using the k-means method. In this case, the distance for k-means is defined by w1 × {(x_i - x) 2 + (y_i - y) 2} 0.5 + w2 × |h_i - h| defined.

[0105] The heat map output unit 318 outputs the heat map clustered by the clustering unit to the display device 400. That is, the heat map (cluster heat map) differentiated in color or shading for each cluster is output to the display device 400.

[0106] Figure 16 This is a flowchart showing the steps (information processing method) of information processing in the information processing system based on this embodiment.

[0107] First, the performer photographing device 100 photographs the performer 1 (step S21). In parallel with this, the viewer photographing device 200 photographs the viewer area 3 and the viewers 2 within the viewer area 3 (step S23).

[0108] Next, the scene of the performance by the performer 1 is recognized based on the image of the performer 1 captured by the performer photographing device 100 (step S22). Also, the faces of the respective audiences 2 are detected from the image of the audience area 3 captured by the audience photographing device 200 (step S24), and the expressions of the detected faces are recognized (step S25). In addition, the popularity of the performer 1 is estimated based on the image of the performer 1 captured by the performer photographing device 100 (step S28), and the number of groups is set according to the estimated popularity (step S29). In the present embodiment, the remaining power related to excitement is estimated as the popularity, and the number of groups is set according to the estimated remaining power.

[0109] Next, the excitement level of each audience 2 corresponding to the scene is calculated based on the recognition result of the scene and the recognition result of the expression (step S26). Next, a heat map is created based on the information on the excitement level of each audience 2 calculated (step S27).

[0110] Next, the heat map is clustered according to the set number of groups (step S30). Thereby, a heat map (clustered heat map) is created that is displayed with colors or shades distinguished for each group.

[0111] Figure 17 It is a conceptual diagram of the change of the heat map based on the clustering process. This figure shows an example when the number of groups is set to 2. (A) is the heat map HM before clustering, and (B) is the heat map (clustered heat map) CHM after clustering. As shown in this figure, after clustering, colors or shades are displayed for each group.

[0112] The created clustered heat map is output and displayed on the display device 400 (step S18). The performer 1 can grasp the excitement state of the audiences in the audience area 3 by visually recognizing the clustered heat map displayed on the display device 400. Since the clustered heat map is displayed with colors or shades for each group, the state of the audiences can be grasped intuitively. Also, since the number of groups is set according to the popularity (here, the remaining power related to excitement) that the performer 1 has, the control of the performance content also becomes easy. That is, for example, when there is no remaining power, clustering is performed with a smaller number of groups, so it is possible to easily determine the content of the performance that should be carried out.

[0113] In addition, in the above-described embodiment, the remaining power that the performer has is used as the popularity, but the skills that the performer has can also be used as the popularity. Also, both the remaining power and the skills can be used as the popularity.

[0114] [Third Embodiment]

[0115] In the information processing system of the present embodiment, when clustering and outputting a heat map, the number of clusters is set according to the spreadability set by a user (e.g., a performer). Except for setting the number of clusters according to the spreadability set by the user, it is the same as the information processing system of the second embodiment described above. Therefore, hereinafter, only the differences will be described.

[0116] Figure 18 It is a block diagram of the functions implemented by the information processing device of the present embodiment.

[0117] As shown in this figure, the information processing device 300 of the present embodiment is provided with a spreadability input unit 324. The spreadability input unit 324 accepts the input of spreadability. As described above, the spreadability is defined by the skills and remaining strength of the performer, etc. The spreadability input unit 324 accepts the input of spreadability from the operation unit 305. For example, for skills, multiple levels are predefined, and one of them is selected to input the spreadability. Similarly, for the remaining strength, multiple levels are also predefined, and one of them is selected to input. The information of the spreadability input to the spreadability input unit 324 is added to the number-of-clusters setting unit 322.

[0118] The number-of-clusters setting unit 322 sets the number of clusters according to the input information of the spreadability. When the spreadability is defined by both skills and remaining strength, the number of clusters is set according to the information of both.

[0119] The clustering unit 323 performs clustering processing on the heat map according to the number of clusters set by the number-of-clusters setting unit 322.

[0120] In the information processing system of the present embodiment, since the heat map is clustered and presented according to the spreadability of the performer, it is also easy to determine the content of the performance to be carried out.

[0121] [Fourth Embodiment]

[0122] In the information processing system of the present embodiment, when clustering and outputting a heat map, the number of clusters is set according to the spreadability set by a user (e.g., a performer) and the spreadability estimated from the image of the performer being photographed.

[0123] Figure 19 It is a block diagram of the functions implemented by the information processing device of the present embodiment.

[0124] As shown in this figure, the information processing device 300 of the present embodiment is provided with a spreadability estimation unit 321 and a spreadability input unit 324.

[0125] The spreadability estimation unit 321 estimates the first spreadability of the performer based on the image of the performer being photographed. In the present embodiment, the remaining strength of the performer is estimated as the first spreadability.

[0126] The spread input unit 324 receives the input of the second spread from a user (e.g., a performer). In the present embodiment, the skill possessed by the performer is used as the second spread, and its input is received.

[0127] The information of the first spread (remaining power) estimated by the spread estimation unit 321 and the information of the second spread (skill) input to the spread input unit 324 are added to the group number setting unit 322.

[0128] The group number setting unit 322 sets the group number based on the information of the first spread (remaining power) and the second spread (skill).

[0129] The clustering unit 323 performs clustering processing on the heat map according to the group number set by the group number setting unit 322.

[0130] In the information processing system of the present embodiment, since the heat map is clustered and presented according to the spread of the performer, it is also easy to determine the content of the performance to be carried out.

[0131] In addition, in the present embodiment, it is configured that the remaining power is estimated by the spread estimation unit 321 and the skill is input by the spread input unit 324, but it may also be configured that the skill is estimated by the spread estimation unit 321 and the remaining power is input by the spread input unit 324.

[0132] [Embodiment 5]

[0133] As described above, a heat map is created by establishing a corresponding association between the excitement level of each audience and the position of each audience. The position of each audience is determined based on the image captured by the audience photographing device 200. The viewpoint of the image captured by the audience photographing device 200 is different from the viewpoint of the performer. In the information processing system of the present embodiment, the image of the heat map is converted into an image with a different viewpoint and presented. That is, it is converted into an image with the viewpoint of the performer (an image close to the viewpoint of the performer) and presented.

[0134] Figure 20 It is a block diagram of the functions implemented by the information processing device of the present embodiment.

[0135] As shown in this figure, the information processing device 300 of the present embodiment further includes an image processing unit 331. Except for including the image processing unit 331, it is the same as the information processing device of the first embodiment described above. Therefore, hereinafter, only matters related to the image processing unit 331 will be described.

[0136] The image processing unit 331 is an example of a conversion processing unit. The image processing unit 331 processes the image of the heat map created by the heat map creation unit 317 to generate a heat map with a changed viewpoint. Specifically, a projective conversion process is performed on the image of the heat map created by the heat map creation unit 317 to convert it into an image from the viewpoint of the performer. Additionally, it is difficult to make it exactly the same as the image from the performer's viewpoint. Therefore, here it is converted into an image close to the performer's viewpoint. Specifically, it is converted into an image from the viewpoint of the performer when standing at a determined position on the stage.

[0137] Figure 21 This is a diagram showing an example of the conversion of the heat map image. This diagram (A) shows the heat map HM0 before the conversion process, and this diagram (B) shows the heat map HM1 after the conversion process.

[0138] In this way, by presenting the heat map according to the image from the performer's viewpoint, the situation can be grasped more easily.

[0139] In addition, the display switching can be configured to be performed according to an instruction from the user (e.g., the performer). In this case, for example, the display switching is performed by the instruction input of the operation unit 305.

[0140] Moreover, this process can also be performed on the heat map of the cluster (cluster heat map).

[0141] [Sixth Embodiment]

[0142] In the information processing system of this embodiment, according to an instruction from the user (e.g., the performer), the heat map is inverted and presented. That is, for example, when the heat map is displayed in color, the color is inverted and presented. And, for example, when the heat map is displayed with the shade of a specific color, the shade of that color is inverted and presented.

[0143] Figure 22 This is a block diagram of the functions implemented by the information processing device of this embodiment.

[0144] As shown in this diagram, the information processing device 300 of this embodiment further includes an image processing unit 331. Except for including the image processing unit 331, it is the same as the information processing device of the above first embodiment. Therefore, hereinafter, only matters related to the image processing unit 331 will be described.

[0145] The image processing unit 331 is an example of an inversion processing unit. The image processing unit 331 processes the image of the heat map according to the input of the inversion instruction from the operation unit 305 to generate a heat map with the color or the shade of the color inverted.

[0146] Figure 23This is a diagram showing an example of the inversion process of an image representing a heat map. This diagram shows an example when creating a heat map based on the shade of a specific color. Diagram (A) shows the heat map HM0 before the inversion process, and diagram (B) shows the heat map HM2 after the inversion process. As shown in this diagram, through the inversion process, the shades are inverted and displayed.

[0147] In this way, by being able to switch the display, the optimal heat map can be presented according to the purpose. For example, in cases such as arousing an audience that is not excited, the heat map is displayed in the normal way. On the other hand, in cases such as making an excited audience even more excited, the inverted heat map is displayed. Thus, it becomes easier to control the content of the performance.

[0148] In addition, this process can also be applied to the heat map of a cluster (cluster heat map).

[0149] [Embodiment 7]

[0150] Figure 24 This is a diagram showing the schematic structure of the information processing system of this embodiment.

[0151] As shown in this diagram, the information processing system 10 of this embodiment is provided with a projection device 500 instead of the display device 400. Except for the fact that a projection device 500 is provided instead of the display device 400, it is the same as the information processing system of the above-mentioned first embodiment. Therefore, hereinafter, only the structure related to the projection device 500 will be described.

[0152] The projection device 500 is an example of a projection unit. The projection device 500 is composed of at least one projector, and projects the heat map (including the cluster heat map) output from the heat map output unit 318 onto the audience area 3 of the audience 2. When the audience area 3 of the audience 2 cannot be projected by one projector, multiple projectors are combined to form it. In this case, the audience area 3 of the audience 2 is divided into multiple areas, and each area is projected by multiple projectors sharing the work.

[0153] According to the information processing system of this embodiment, since the heat map is projected onto the audience area 3 of the audience 2, the excitement state of the audience can be grasped at a glance. And since there is no need to confirm the display device, the performer can focus on the performance. Also, the audience notices their own level of excitement.

[0154] In addition, usually, the audience area 3 of the audience 2 is not flat, so it is preferable to use the method of projection mapping for projection.

[0155] [Other Embodiments]

[0156] [Regarding Scene Recognition]

[0157] In the above-described embodiment, the performance scene is recognized based on the image of the performer being photographed, but other information can also be used to recognize the performance scene. For example, it can also be configured to recognize the scene using sound or information of sound and image. In this case, a sound collection unit for collecting the sound accompanying the performance is further provided.

[0158] [Regarding the degree of excitement]

[0159] In the above-described embodiment, the degree of excitement is calculated based on the expressions of the audience, but other information can also be used to calculate the degree of excitement. For example, information on the sounds emitted by each audience, information on body swaying, information on body temperature, etc. can also be used to calculate the degree of excitement. In this case, a sound collection unit for collecting the sounds emitted by each audience, a sensor for detecting the swaying of each audience's body, a sensor for detecting the body temperature of each audience, etc. are provided. And an input unit for inputting this information is provided.

[0160] [Regarding the estimation of spreadability]

[0161] In the above-described embodiment, the spreadability possessed by the performer is estimated based on the image of the performer being photographed, but other information can also be used to estimate the spreadability. For example, it can also be configured to estimate the spreadability based on the sound (volume, sound quality, etc.) emitted by the performer. And it can also be configured to estimate the spreadability based on both the sound emitted by the performer and the image of the performance. In this case, for example, the performer photographing device 100 photographs a video with sound.

[0162] And it can also be configured to estimate the performer's skill based on the state of the audience. That is, since the degree of excitement of the audience also varies depending on the skill, the skill possessed by the performer can be estimated based on the degree of excitement of the audience. In this case, for example, it can be configured to estimate the performer's skill based on the information on the degree of excitement of each audience calculated by the degree of excitement calculation unit 316 or the heat map produced by the heat map production unit 317.

[0163] And generally speaking, the ability related to excitement depends to a large extent on the experience of the performer. Therefore, the experience value (number of years of experience, number of times of implementation of activities, etc.) of the performer can also be included in the spreadability.

[0164] [Regarding the display of the heat map]

[0165] It can also be configured to display the heat map on a display device (so-called wearable device) used by being mounted on the performer. For example, it can also be configured to display the heat map on a watch-type or glasses-type display device.

[0166] And it can be configured to also present the heat map to the audience. For example, it can be configured to transmit the information of the heat map to the mobile terminal (for example, a smart phone, etc.) held by the audience.

[0167] [Regarding Information Processing Device]

[0168] Part or all of the functions of an information processing device can be implemented by various processors. The various processors include a general-purpose processor that executes a program and functions as various processing units, namely a CPU (Central Processing Unit), a processor such as an FPGA (Field Programmable Gate Array) that can change the circuit structure after manufacturing, i.e., a programmable logic device (PLD), and a processor with a circuit structure specifically designed for executing specific processing, i.e., an application-specific integrated circuit (ASIC), etc. The meaning of a program is the same as that of software.

[0169] One processing unit can be constituted by one of these various processors, or by two or more processors of the same or different types. For example, one processing unit can also be constituted by a plurality of FPGAs or a combination of a CPU and an FPGA. Also, one processor can constitute multiple processing units. As an example of one processor constituting multiple processing units, first, there is the following method: as represented by a computer such as a client or a server, a combination of one or more CPUs and software constitutes one processor, and this processor functions as multiple processing units. Second, there is the following method: as represented by a system on chip (SoC), etc., a processor that uses one IC (Integrated Circuit) chip to implement the functions of the entire system including multiple processing units is used. Thus, the various processing units are constituted by using one or more of the above various processors as a hardware structure.

[0170] Symbol Explanation

[0171] 1 - Performer, 2 - Audience, 3 - Audience area, 10 - Information processing system, 100 - Performer photography device, 200 - Audience photography device, 300 - Information processing device, 301 - CPU, 302 - ROM, 303 - RAM, 304 - HDD, 305 - Operation unit, 307 - Input interface, 308 - Output interface, 311 - First image input unit, 312 - Second image input unit, 313 - Scene recognition unit, 314 - Face detection unit, 315 - Expression recognition unit, 316 - Intensity calculation unit, 317 - Heat map creation unit, 318 - Heat map output unit, 321 - Spreadability estimation unit, 322 - Group number setting unit, 323 - Clustering unit, 324 - Spreadability input unit, 331 - Image processing unit, 400 - Display device, 500 - Projection device, F - Frame enclosing the detected face, HM - Heat map, HMO - Heat map, HM1 - Heat map after conversion processing, HM2 - Heat map after inversion processing, I2 - Image of the audience, IF - Image of the face of the audience, S11 to S18 - Steps of information processing based on the information processing system, S21 to S31 - Steps of information processing based on the information processing system.

Claims

1. An information processing system, comprising: a processor; and a memory connected to the processor, wherein the processor is configured to: perform a first recognition of identifying a scene based on an image of a performer; perform a second recognition of identifying an expression of a person within a specified area of an audience area based on an image of the person; calculate a condition index of the person corresponding to the scene based on the recognition result of the scene and the recognition result of the expression of the person; create a heat map representing the correspondence between the condition index of the person and the position of the person within the area; output the heat map.

2. The information processing system according to claim 1, wherein the processor is further configured to: set the number of groups based on the influence power of the performer; cluster the heat map according to the set number of groups; in the output, output the clustered heat map.

3. The information processing system according to claim 2, wherein the processor is further configured to: in the clustering, cluster the heat map using the data of the person whose condition index is above a threshold value as an object.

4. The information processing system according to claim 2 or 3, wherein the processor is further configured to: estimate the influence power of the performer based on an image of the performer.

5. The information processing system according to claim 2 or 3, wherein the influence power includes at least one of the skills and spare capacity of the performer.

6. The information processing system according to any one of claims 1 to 3, wherein the processor is further configured to: convert the image of the heat map into an image with a viewpoint different from that of the image of the person within the area.

7. The information processing system according to any one of claims 1 to 3, wherein the processor is further configured to: invert the color or shading of the heat map.

8. The information processing system according to any one of claims 1 to 3, wherein the information processing system further includes a display, and the display displays the heat map output from the processor.

9. The information processing system according to any one of claims 1 to 3, wherein the information processing system further includes a projector, and the projector projects the heat map output from the processor onto the area.

10. The information processing system according to claim 9, wherein the projector projects the heat map onto the area by projection mapping.

11. The information processing system according to any one of claims 1 to 3, wherein the information processing system further includes: a first camera that photographs the performer; and a second camera that photographs the person within the area.

12. An information processing method, including the following steps: photograph a performer; photograph a person within a specified area of an audience area; identify a scene based on an image of the performer; identify an expression of the person based on an image of the person within the area; Calculate the condition index of the person corresponding to the scene based on the recognition result of the scene and the recognition result of the expression of the person. Create a heat map that represents the correspondence between the condition index of the person and the position of the person in the area. Output the heat map.

13. The information processing method according to claim 12, comprising the following steps: Set the number of groups according to the information on the dissemination power of the performer. Cluster the heat map according to the set number of groups. In the output of the heat map, output the clustered heat map.

Images