Information processing device

Abstract

To provide an information processing device capable of setting an optimum travel route according to a situation.SOLUTION: A guiding method includes: a step of capturing an image; a step of detecting a person from the captured image; a step of quantifying and estimating a state of the detected person; a step of creating map data representing the state of the person on a map in accordance with the location of the person in the captured image; a first area estimation unit for estimating a first area, based on the map data; a step of setting a travel route for a person in the first area and / or a person in a second area other than the first area; a step of transmitting information about the set travel route; and a step of receiving the transmitted information about the travel route and presenting the received information about the travel route and / or route guidance information based on the information about the travel route.SELECTED DRAWING: Figure 22

Description

[Technical field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] Patent Document 1 describes a system that provides information on the optimal evacuation route to spectators in each seat in the event of a disaster at an event venue. In Patent Document 1, the system determines the evacuation route from each seat in advance, making it possible to provide the optimal evacuation route to spectators in each seat. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2003-345999 A Summary of the Invention

[0004] One embodiment of the technology disclosed herein provides an information processing device capable of setting a travel route depending on a situation. [Means for solving the problem]

[0005] (1) An information processing device comprising: a person detection unit that detects a person from an input image; an estimation unit that quantifies and estimates the state of the person detected by the person detection unit; a map data creation unit that creates map data that represents the state of the person on a map according to the position of the person in the input image; a first area estimation unit that estimates a first area based on the map data; and a movement path setting unit that sets a movement path of a person in the first area and / or a person in a second area other than the first area.

[0006] (2) The information processing device of (1), further comprising a transmission unit that transmits information about the travel route set by the travel route setting unit.

[0007] (3) An information processing device according to (1) or (2), wherein the estimation unit includes a recognition unit that recognizes at least one of the facial expression and the body movement of the person from the image of the person detected by the person detection unit, and quantifies and estimates the state of the person based on the recognition result of the recognition unit.

[0008] (4) An information processing device according to (3), wherein the estimation unit assigns a score to the recognition result of the recognition unit in accordance with predetermined criteria, thereby quantifying the state of the person.

[0009] (5) An information processing device according to (3) or (4), wherein the recognition unit recognizes the facial expression, facial direction, and body movement of the person from the image of the person detected by the person detection unit, and the estimation unit assigns scores to the recognition results of the facial expression, facial direction, and body movement of the person by the recognition unit in accordance with predetermined criteria, calculates the sum of the scores, and quantifies the state of the person.

[0010] (6) An information processing device according to any one of (1) to (5), wherein the first area estimation unit includes a counting unit that divides an input image into a plurality of partial areas and counts the number of people whose state is equal to or greater than a first threshold for each partial area, and an extraction unit that extracts partial areas in which the number of people whose state is equal to or greater than the first threshold is equal to or greater than a second threshold, and estimates the partial area extracted by the extraction unit as the first area.

[0011] (7) An information processing device according to any one of (1) to (6), further comprising a recording unit which records the state of the person estimated by the estimation unit over time, and a maximum value detection unit which detects a maximum value of the state of the person recorded over time in the recording unit, wherein the map data creation unit creates map data using the maximum value.

[0012] (8) Any one of the information processing devices (1) to (7), wherein the input image is an image taken of a venue having a seating area in which a plurality of seats are arranged, and a first area and a second area are set within the seating area.

[0013] (9) An information processing device according to any one of (1) to (8), further comprising an output unit that outputs visualized data of map data and / or information on a travel route.

[0014] (10) An information processing device according to any one of (1) to (9), further comprising a display unit that displays visualized map data and / or information on a travel route.

[0015] (11) A guidance system comprising an information processing device as defined in (2), an imaging device for capturing images to be input to the information processing device, and a presentation device for receiving information on a travel route transmitted from the information processing device and presenting the received information on the travel route and / or route guidance information based on the information on the travel route.

[0016] (12) An information processing method including the steps of: detecting a person from an input image; quantifying and estimating the state of the detected person; creating map data that represents the state of the person on a map according to the person's position in the input image; a first area estimation step of estimating a first area based on the map data; and setting a movement route of the person in the first area and / or a person in a second area other than the first area.

[0017] (13) The information processing method according to (12), further comprising the step of transmitting information about the set travel route.

[0018] (14) A guidance method including the steps of: capturing an image; detecting a person from the captured image; quantifying and estimating the state of the detected person; creating map data that represents the state of the person on a map according to the position of the person in the captured image; a first area estimation step of estimating a first area based on the map data; setting a movement route for a person in the first area and / or a person in a second area other than the first area; transmitting information of the set movement route; receiving the transmitted movement route information and presenting route guidance information based on the received movement route information and / or the movement route information. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 shows a schematic configuration of a guidance system. [Diagram 2] A diagram showing an example of an event venue [Diagram 3] Diagram showing the layout of the connecting passages [Figure 4] A diagram showing an example of camera shooting range settings [Diagram 5] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing device. [Figure 6] Block diagram of functions realized by the information processing device [Figure 7] Block diagram of the functions of the estimation unit [Figure 8] Conceptual diagram of the process from detecting people to recognizing facial expressions, facial orientation, and body movements. [Figure 9] FIG. 1 shows an example of a judgment criterion [Figure 10] A diagram showing an example of visualized map data. [Figure 11] A diagram showing the relationship between excitement level and dot display density [Figure 12] A diagram showing an example of dividing the spectator area [Figure 13] Block diagram of the functions of the danger area estimation unit [Figure 14] FIG. 13 is a diagram showing an example of the excitement level of each spectator in a certain partial area. [Figure 15] FIG. 13 is a diagram showing an example of a count result by a counting unit. [Figure 16] FIG. 13 is a diagram showing an example of setting a movement route when there is no dangerous area within the spectator area. [Figure 17] FIG. 13 is a diagram showing an example of setting a movement route when a dangerous area exists within a spectator area. [Figure 18] FIG. 1 is a diagram showing an example of an installation position of a guidance display device; [Figure 19] FIG. 1 is a block diagram showing a configuration of a guidance display device; [Figure 20] Block diagram of functions of a display control unit [Figure 21] FIG. 13 is a diagram showing an example of guidance information displayed on a display; [Figure 22] Flowchart showing the process flow of the guidance system [Figure 23]Block diagram of functions realized by the information processing device [Figure 24] A flowchart showing the procedure of processing performed by the information processing device. [Diagram 25] FIG. 11 is a diagram showing another example of a method for presenting guidance information; [Figure 26] FIG. 11 is a diagram showing another example of a method for presenting guidance information; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0021] [First embodiment] When guiding spectators at an event venue, it is common to select and guide them along the shortest route to the destination. However, it is not always appropriate to uniformly guide all spectators along the shortest route. For example, if there is a group of excited spectators in the venue, guiding other spectators along the same route as these excited spectators may cause the other spectators to get into unexpected trouble. In this embodiment, a guidance system is provided that can set the movement route of people such as spectators according to the situation.

[0022] [Guidance system configuration] Here, an example will be described in which spectators are guided within an event venue having seats (spectator seats).

[0023] FIG. 1 is a diagram showing a schematic configuration of a guidance system according to the present embodiment.

[0024] As shown in the figure, the guidance system 1 of this embodiment comprises a camera 10 that captures images of the venue, an information processing device 100 that processes images captured by the camera 10 and sets the movement route of spectators within the venue, and a guidance display device 200 that displays route guidance information based on information about the movement route set by the information processing device 100.

[0025] [Photographic equipment] The image capturing device 10 captures an area in an event venue where spectators are present (hereinafter referred to as the spectator area). In an event venue having seats (spectator seats), the area in which the seats are arranged (seat area) corresponds to the spectator area. Therefore, in an event venue having seats, the seat area is captured as the spectator area.

[0026] The photographing device 10 is composed of a camera (a so-called digital camera, including a digital video camera) that records images as digital data. When it is not possible to photograph all spectator areas in a venue with a single camera, the photographing device 10 is composed of a combination of multiple cameras. In this case, the spectator area is divided into multiple areas, and each area is assigned to photograph. Note that when it is not possible to photograph all spectator areas with a single camera, it refers to a case where it is not possible to determine the state of each spectator in the spectator area from the photographed image, and in particular, when it is not possible to determine the facial expressions. Therefore, even if all spectator areas are within the angle of view, when it is not possible to determine the state of the spectators from the photographed image, this corresponds to a case where it is not possible to photograph with a single camera.

[0027] 2 is a diagram showing an example of an event venue, showing a plan view of the event venue.

[0028] The event venue 300 shown in FIG. 2 is a so-called sports ground (stadium). The event venue 300 has a ground 310 where athletes compete and a spectator area 320 where spectators watch the competition. The ground 310 has a rectangular shape, and the spectator area 320 is provided in a manner surrounding the ground. The spectator area 320 has a mortar-shaped (concave shape in a pyramid shape) structure, and the seats (spectator seats) are arranged in a stepped manner. The spectator area 320 has a two-story structure, and has a first floor seating area 322 and a second floor seating area 324. Between the first floor seating area 322 and the second floor seating area 324, a circular passage 326 (hereinafter referred to as the circular passage 326) that forms a landing is provided. The first floor seating area 322 and the second floor seating area 324 are each divided into a plurality of blocks in the circumferential direction. In the first floor seating area 322 and the second floor seating area 324, seats are arranged in blocks. The seats are regularly arranged in each block. Aisles 328 (hereinafter referred to as inter-block passages 328) are provided between each block. The inter-block passages 328 are connected to the circular passage 326. The inter-block passage 328 is provided with handrails 328A in the center in the width direction (see FIG. 4). The handrails 328A separate the inter-block passage 328 into left and right. The handrails 328A are arranged at regular intervals along the inter-block passage 328. The circular passage 326 is provided with entrances 332X of passages 332 (hereinafter referred to as connecting passages 332) connected to the outer circumferential area 330 at multiple locations.

[0029] FIG. 3 is a diagram showing the arrangement of the communication passages.

[0030] The outer periphery area 330 is provided as a ring-shaped area on the outer periphery of the spectator area 320. The outer periphery area 330 is provided with gates 334A-334D at four locations around it. Spectators enter the event venue 300 using any of the gates 334A-334D. Spectators who have entered the event venue 300 enter the spectator area 320 from the outer periphery area 330 through any of the connecting passages 332. In the spectator area 320, spectators pass through the ring-shaped passage 326 and the inter-block passage 328 to reach their seats.

[0031] As shown in Fig. 2, the photographing device 10 of this embodiment is configured by combining multiple cameras 12. Each camera 12 photographs a target area at a fixed position. Fig. 4 is a diagram showing an example of the setting of the photographing range of each camera. As shown in the figure, the photographing ranges W of cameras photographing adjacent areas are set to overlap in part. This allows the spectator area 320 to be photographed without omission.

[0032] Each camera 12 is installed in a position where it can capture the faces of spectators in the target area. Each camera 12 is also configured with the ability to recognize the facial expressions of spectators from the captured images. In other words, each camera is provided with a necessary and sufficient resolution performance to recognize the facial expressions of spectators from the captured images.

[0033] The photographing device 10 synthesizes the images captured by each camera 12 and outputs (including transmission) to the information processing device 100. That is, a single image capturing all of the spectator areas 320 is generated and output to the information processing device 100. When synthesizing and outputting images in this manner, the photographing device 10 is provided with a synthesis processing unit. In this case, the photographing device 10 is configured with a plurality of photographing units (cameras) and a synthesis processing unit that synthesizes the images captured by each photographing unit. The synthesis processing can also be configured to be performed by a device other than the photographing device 10. For example, it can be configured to be performed by the information processing device 100.

[0034] [Information processing device] The information processing device 100 processes the images captured by the image capturing device 10 to set a route for visitors within the venue, and transmits the route to the guide display device 200.

[0035] FIG. 5 is a block diagram illustrating an example of a hardware configuration of an information processing device.

[0036] The information processing device 100 is composed of a computer equipped with a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a HDD (Hard Disk Drive) 104, an operation unit (e.g., a keyboard, a mouse, a touch panel, etc.) 105, a display unit (e.g., a liquid crystal display (LCD), an organic electro-luminescence display (OLED), etc.) 106, an image input unit 107, and a communication unit 108. An image captured by the imaging device 10 is input (including reception) to the information processing device 100 via the image input unit 107. Information on the travel route in the information processing device 100 is transmitted to the guidance display device 200 via the communication unit 108.

[0037] FIG. 6 is a block diagram of functions realized by the information processing device.

[0038] As shown in the figure, the information processing device 100 realizes the functions of an image acquisition unit 110 that acquires an image captured by the image capture device 10, a person detection unit 112 that detects a person from the image acquired by the image acquisition unit 110, an estimation unit 114 that estimates the state of the person detected by the person detection unit 112, a map data creation unit 116 that creates map data that shows the state of each spectator in the venue on a map according to the position of each spectator based on the estimation result of the estimation unit 114, a danger area estimation unit 118 that estimates danger areas in the venue based on the map data, a movement route setting unit 120 that sets a movement route for the spectators based on information on the danger area estimated by the danger area estimation unit 118, and a transmission unit 122 that transmits information on the set movement route to the guidance display device 200. Each function is realized by a CPU 101, which is a processor, executing a predetermined program.

[0039] The image acquisition unit 110 imports an image (captured image) output from the photographing device 10 into the information processing device 100 via the image input unit 107. The imported image (image captured by the photographing device 10) constitutes an input image.

[0040] The person detection unit 112 detects a person from the input image. Specifically, the person is detected by detecting the face of the person from the image. This type of technology is well known, so a detailed description of the method will be omitted. For example, a method of detecting a person from an image using an image recognition model generated by machine learning, deep learning, or the like can be adopted. A person is detected by identifying a position (a position within the input image). The position within the input image is identified, for example, by coordinates set for the input image. In the information processing device 100 of this embodiment, the input image is an image captured of the spectator area. Therefore, the person detection unit 112 detects spectators (people) in the spectator area.

[0041] The estimation unit 114 estimates the state of a person (audience) detected by the person detection unit 112 from an image of the person. In this embodiment, the degree of excitement is estimated from the image as the state of the person. The degree of excitement is estimated by quantifying it. The degree of excitement quantified is defined as the degree of excitement. In this embodiment, the degree of excitement is calculated by the following method. First, each of the facial expression, face direction, and body movement of the person is recognized from the image of the person. Next, a score is assigned to each recognized item according to a predetermined judgment criterion. The judgment criterion has a configuration in which the higher the score is assigned to an item that is considered to be more exciting for each item. Finally, the sum of the assigned scores is calculated. The sum of the calculated scores becomes the estimated result of the excitement level of the person.

[0042] FIG. 7 is a block diagram of functions possessed by the estimation unit.

[0043] As shown in the figure, the estimation unit 114 has the functions of an expression recognition unit 114A that recognizes a person's facial expression from an image of the person, an orientation recognition unit 114B that recognizes the orientation of the face from the image of the person, a movement recognition unit 114C that recognizes body movement from the image of the person, a scoring unit 114D that scores the items recognized by each recognition unit, and an excitement level calculation unit 114E that calculates the excitement level by adding up the scoring results.

[0044] The facial expression recognition unit 114A recognizes the facial expression of a person (audience) detected by the person detection unit 112 from an image of the person, particularly an image of the face. In this embodiment, the facial expression of the target is recognized by identifying one plausible one from among a plurality of types of facial expressions determined in advance. The type of facial expression can be expressed by a word indicating an emotion. In this embodiment, facial expressions are classified into seven types, namely, "happiness," "anger," "disgust," "surprise," "fear," "sadness," and "serious face (expressionless)," and the facial expression is recognized by identifying one plausible one. The technology for recognizing a person's facial expression from an image is a known technology. In this embodiment, a known method is also adopted to recognize a person's facial expression from an image. For example, a method of recognizing a person's facial expression from an image using an image recognition model generated by machine learning, deep learning, or the like can be adopted. Note that, as described above, the type of facial expression can be expressed by a word indicating an emotion. Therefore, recognizing a facial expression is synonymous with recognizing an emotion.

[0045] The direction recognition unit 114B recognizes the direction of the face of a person (spectator) detected by the person detection unit 112 from an image of the person, particularly an image of the face. That is, it recognizes the direction in which the face is facing (front, up, down, sideways, diagonally upward, diagonally downward). Publicly known technology can be used for this process as well.

[0046] The movement recognition unit 114C recognizes the body movement of a person (spectator) detected by the person detection unit 112 from the image of the person. In this embodiment, it recognizes whether or not the person is making a predetermined movement. For example, it recognizes whether the person is standing, sitting, jumping, raising his / her hands, clapping his / her hands, etc. Publicly known technology can be used for this process as well.

[0047] FIG. 8 is a conceptual diagram of the process from human detection to recognition of facial expressions, face orientation, and body movements.

[0048] As shown in the figure, a person (spectator) is detected from the captured image, and the facial expression, face direction, and body movement of the detected person are recognized from the image IP of the person. Note that in the figure, the image indicated by the symbol IS1 is an image showing a part of the captured image, and the image indicated by the symbol IS2 is an enlarged image of a part of the image IS1 (the area surrounded by a rectangular frame).

[0049] Scoring section 114D scores the items recognized by each recognition section (facial expression recognition section 114A, direction recognition section 114B, and movement recognition section 114C) by assigning points according to predetermined criteria.

[0050] FIG. 9 is a diagram illustrating an example of the determination criteria.

[0051] As shown in the figure, a score is assigned to each item recognized by each recognition unit. The higher the score, the more excited the item is considered to be. Information on the judgment criteria is stored, for example, in ROM 103 and HDD 104. The scoring unit 114D assigns a score to each item recognized by each recognition unit according to the judgment criteria. For example, if the recognition result of the "facial expression" of a person (spectator) is "anger," the recognition result of the "face direction" is "front," and the recognition result of the "body movement" is "jumping," the score assigned to the "facial expression" item is 5 points (anger), the score assigned to the "face direction" item is 5 points (front), and the score assigned to the "body movement" item is 5 points (jumping). For example, if the recognition result of a person's (spectator's) "facial expression" is "sadness," the recognition result of the "face direction" is "down," and the recognition result of the "body movement" is "sitting," the score assigned to the "facial expression" item will be 1 point (sadness), the score assigned to the "face direction" item will be 1 point (down), and the score assigned to the "body movement" item will be 1 point (sitting).

[0052] The excitement level calculation unit 114E calculates the excitement level by adding up the scores given by the scoring unit 114D. For example, if a person (spectator) is given 5 points for the "facial expression", 5 points for the "face direction", and 5 points for the "body movement", the excitement level of the person is calculated as 15 (=5+5+5).

[0053] The excitement level calculated by the excitement level calculation unit 114E is added to the map data creation unit 116 as an estimated result of the state of the person. Information on the excitement level of each spectator is associated with information on the position of each spectator and added to the map data creation unit 116.

[0054] Based on the estimation result of the estimation unit 114, the map data creation unit 116 creates map data that shows the excitement level (state of the audience) of each audience member in the venue on a map according to the position of each audience member. The map is a map of the event venue. As described above, the excitement level of each audience member is calculated with the position in the captured image specified. The map data creation unit 116 converts the position of each audience member in the captured image to a position on the map to specify the position of each audience member on the map. The position conversion is performed, for example, by using a conversion formula or the like. The conversion formula or the like is generated by determining the relationship between the captured image and the map in advance. The generated conversion formula or the like is stored in the HDD 104. The position on the map is specified, for example, by coordinates set on the map. In this case, the coordinate position of each audience member in the captured image is converted to a coordinate position on the map.

[0055] The map data can be visualized by expressing the excitement level of each spectator with a color or a shade of gray. That is, the excitement level of each spectator can be visualized on the map image by expressing the color or shade of gray according to the excitement level value (so-called heat map).

[0056] 10 is a diagram showing an example of visualized map data, which is a part of the visualized map data (map image).

[0057] In the example shown in the figure, the positions of the spectators are indicated by dots (circular points) dp on the map image MI. The positions of the dots dp correspond to the positions of the spectators in the captured image. The positions of the spectators roughly coincide with the positions of the seats. The dots dp indicating the positions of the spectators are displayed with a density according to the excitement level of the spectators. FIG. 11 is a diagram showing the relationship between the excitement level and the display density of the dots. The dots dp are displayed with a darker density the higher the excitement level numerical value. Therefore, the spectators displayed with darker dots dp are more excited. From the example shown in FIG. 10, it can be seen that highly excited spectators are concentrated in the block diagonally upper left in the figure.

[0058] The danger area estimation unit 118 estimates danger areas within the venue based on the map data created by the map data creation unit 116. A danger area is an area in the spectator area 320 that is estimated to be dangerous. In this embodiment, an area in which there is a certain number or more highly excited spectators is considered to be a danger area. The danger area is an example of a first area. Moreover, the danger area estimation unit 118 is an example of a first area estimation unit.

[0059] The risk area is estimated, for example, as follows: The spectator area 320 is divided into a plurality of partial areas. The number of spectators whose excitement level is equal to or greater than a first threshold is counted for each divided partial area. After the number of spectators whose excitement level is equal to or greater than the first threshold is counted for all partial areas, partial areas in which the number of spectators whose excitement level is equal to or greater than the first threshold is equal to or greater than a second threshold are extracted. The extracted partial areas are estimated to be risk areas.

[0060] Fig. 12 is a diagram showing an example of the division of the spectator area. In the example shown in the figure, almost all of the seat blocks are divided into two equal parts in the circumferential direction (the direction in which the seats are arranged horizontally), dividing the spectator area 320 into a number of partial areas BA.

[0061] FIG. 13 is a block diagram of functions possessed by the risk area estimation unit.

[0062] As shown in the figure, the danger area estimation unit 118 has a counting unit 118A that counts the number of people (spectators) whose excitement level is equal to or greater than a first threshold for each partial area, and a danger area extraction unit 118B that extracts partial areas in which the number of people (spectators) whose excitement level is equal to or greater than the first threshold is equal to or greater than a second threshold as danger areas.

[0063] The counting unit 118A counts the number of people (audience) whose excitement level is equal to or higher than the first threshold for each partial area based on the division information of the spectator area 320. The division information is information indicating how to divide the spectator area 320. The division information of the spectator area 320 and the first threshold are stored in, for example, the ROM 103 and the HDD 104.

[0064] Fig. 14 is a diagram showing an example of the excitement level of each audience member in a certain partial area. In this example, the first threshold is set to 11. In this case, the number of audience members whose excitement level is 11 or more is counted. In the example shown in Fig. 14, there are 15 audience members.

[0065] The dangerous area extraction unit 118B extracts partial areas in which the number of people (audience) whose excitement level is equal to or greater than the first threshold is equal to or greater than the second threshold based on the counting result by the counting unit 118A. The dangerous area extraction unit 118B is an example of an extraction unit. The second threshold can be a common value for each partial area, or can be determined for each partial area. For example, if the number of people constituting each partial area (the number of seats if there are seats) differs, the second threshold can be set for each partial area. In this case, for example, the number of people at a predetermined ratio to the number of people constituting the partial area can be set as the second threshold. The second threshold is stored in, for example, the ROM 103 or the HDD 104.

[0066] Fig. 15 is a diagram showing an example of the counting result by the counting unit. In the figure, the numerical value in each partial area is the number of people (spectators) whose excitement level is equal to or higher than the first threshold in each partial area. The second threshold is uniformly set to 20 people in each partial area. In the example shown in Fig. 15, three partial areas indicated by white numbers are extracted as dangerous areas.

[0067] The movement route setting unit 120 sets a movement route for spectators based on information about the danger area estimated by the danger area estimation unit 118. In the information processing device 100 of this embodiment, a route that guides spectators in the spectator area 320 to the outer perimeter area 330 is set as the movement route. The movement route setting unit 120 sets movement routes separately for spectators in danger areas and spectators in areas other than the danger areas (non-danger areas). The non-danger areas are areas of the spectator area 320 excluding the danger areas. The non-danger areas are an example of a second area.

[0068] As described above, spectators travel between the spectator area 320 and the outer perimeter area 330 through the connecting passage 332. Therefore, the movement path setting unit 120 sets paths (movement paths) that lead spectators in danger areas and non-danger areas to the connecting passage 332. When setting the paths, the movement path setting unit 120 sets the paths so that the paths of spectators in danger areas do not intersect with the paths of spectators in non-danger areas. In this embodiment, the paths for each area are set as follows, so that the paths of spectators in danger areas do not intersect with the paths of spectators in non-danger areas.

[0069] First, a method for setting a movement route when there is no dangerous area within the spectator area 320 will be described.

[0070] FIG. 16 is a diagram showing an example of setting a movement route when there is no dangerous area within the spectator area.

[0071] In the figure, a route R1 indicated by an arrow is a movement route set for each of the partial areas BA1 to BA12. In this case, the shortest route from each of the partial areas BA1 to BA12 to the nearest communication passage 332 is set as the movement route for each of the partial areas BA1 to BA12. For example, in Figure 16, for the first partial area BA1, the first communication passage 332A is the nearest communication passage, so the shortest route to the first communication passage 332A is set as the movement route.

[0072] Next, a method for setting a movement route when a danger area exists within the spectator area 320 will be described.

[0073] 17 is a diagram showing an example of setting a movement route when a danger area exists within a spectator area. The figure shows an example in which a fourth partial area BA4 is estimated to be a danger area.

[0074] First, the nearest connecting passage is identified from each of the partial areas BA1 to BA12. The shortest route to the identified connecting passage is set (provisionally set) as the movement route of each of the partial areas BA1 to BA12. Next, partial areas other than the dangerous area (non-dangerous areas) that use the same connecting passage as the dangerous area are extracted. In the example shown in FIG. 17, the third partial area BA3, the ninth partial area BA9, and the tenth partial area BA10 in the figure are partial areas that use the same connecting passage (the second connecting passage 332B) as the fourth partial area BA4, which is a dangerous area. Next, the movement route of the partial area that uses the same connecting passage as the dangerous area is reset. At this time, a route that leads to a connecting passage different from the dangerous area is set. Specifically, the next nearest connecting passage is searched for, and the shortest route to the searched connecting passage is set as the movement route. For example, in the example shown in FIG. 17, the third partial area BA3 has the first connecting passage 332A as the next closest connecting passage, so the shortest route to the first connecting passage 332A is set as the movement route of the third partial area BA3. Similarly, the ninth partial area BA9 has the first connecting passage 332A as the next closest connecting passage, so the shortest route to the first connecting passage 332A is set as the movement route of the ninth partial area BA9. Also, the tenth partial area BA10 has the third connecting passage 332C as the next closest connecting passage, so the shortest route to the third connecting passage 332C is set as the movement route of the tenth partial area BA10. By setting in this way, it is possible to set the movement route of spectators in non-dangerous areas while avoiding dangerous areas. In other words, it is possible to set a highly safe movement route for spectators in non-dangerous areas.

[0075] It is also possible that the next nearest connecting passageway will be the same as the one used by spectators in the danger area. In this case, the shortest route to the next nearest connecting passageway will be set as the travel route. However, setting it in this way may result in a travel route being set that leads to an extremely distant connecting passageway. If a suitable connecting passageway (a connecting passageway that does not lead to the same spectators as those in the danger area) cannot be found (searched) within a predetermined distance range (within a threshold), the shortest route to the nearest connecting passageway will be set as the travel route.

[0076] As shown in Fig. 6, the transmitting unit 122 transmits information about the movement route to the guidance display device 200 via the communication unit 108. As will be described later, a guidance display device 200 is provided for each partial area. Therefore, it is also possible to configure the system so that only information about the movement route of the corresponding partial area is transmitted individually to each guidance display device 200. Alternatively, it is also possible to configure the system so that information about the movement routes of all spectator areas is transmitted uniformly to each guidance display device 200.

[0077] [Guide display device] As shown in Fig. 1, the guidance display device 200 receives information on a travel route transmitted from the information processing device 100, and generates and displays route guidance information based on the received travel route information. The guidance display device 200 is an example of a presentation device. The guidance display device 200 is installed at multiple locations within the venue.

[0078] FIG. 18 is a diagram showing an example of the installation position of the guidance display device.

[0079] As shown in the figure, in the guidance system 1 of this embodiment, a guidance display device 200 is installed at the intersection of the circular passage 326 and the inter-block passage 328. A guidance display device 200 is provided for each partial area. In the example shown in FIG. 18, the guidance display device 200 arranged on the opposite side across the circular passage 326 becomes the guidance display device 200 for each partial area. For example, in FIG. 18, the guidance display device indicated by the reference symbol BAi corresponds to the guidance display device indicated by the reference symbol 200i.

[0080] FIG. 19 is a block diagram showing the configuration of the guidance display device.

[0081] The guidance display device 200 includes a display 210 and a display control unit 220 that controls the display of the display 210 .

[0082] The display 210 is a display unit for displaying guidance information. The display 210 is configured with, for example, an LCD, an OELD, or the like.

[0083] The display control unit 220 receives information about a travel route transmitted from the information processing device 100. In addition, based on the received information about the travel route, the display control unit 220 generates guidance information and displays it on the display 210. The display control unit 220 is configured with a computer having a function of communicating with the information processing device 100.

[0084] FIG. 20 is a block diagram of functions possessed by the display control unit.

[0085] As shown in the figure, the display control unit 220 has the functions of a travel route information receiving unit 220A, a guidance information generating unit 220B, and a display control unit 220C. These functions are realized by a CPU constituting a computer executing a predetermined program.

[0086] The travel route information receiving unit 220A receives information on the travel route transmitted from the information processing device 100. Note that the form of communication between the information processing device 100 and the guidance display device 200 is not particularly limited.

[0087] The guidance information generating unit 220B generates guidance information based on the received information on the travel route, at least for a corresponding partial area.

[0088] The display control unit 220C causes the display 210 to display the generated guidance information.

[0089] FIG. 21 is a diagram showing an example of guidance information displayed on a display.

[0090] In the example shown in the figure, a map In1 of the surroundings of the current location (the point where the guidance display device 200 is installed) and an image In showing an arrow In2 indicating the travel direction are displayed as guidance information. The set travel route is displayed with arrows on the surrounding area map In1.

[0091] [Guidance system operation] FIG. 22 is a flowchart showing the flow of processing of the guidance system of this embodiment.

[0092] [Processing at the imaging device] First, photography is performed by the photographing device 10 installed at the event venue (step S11). An image (photographed image) photographed by the photographing device 10 is output to the information processing device 100 (step S12). This image is an image of the spectator area.

[0093] [Processing by information processing device] The information processing device 100 receives an image (captured image) output from the image capturing device 10 (step S21). The information processing device 100 sets a movement route of the spectators based on the input captured image (input image). The movement route is set in the following procedure.

[0094] First, people (spectators) in the spectator area are detected from the captured image (step S22). The positions of the people in the image are identified and detected.

[0095] Next, the excitement level (state of the person) of the detected person is estimated (step S23). The excitement level is estimated from the image. Specifically, facial expressions, facial orientation, and body movements are recognized from the image, and the excitement level is estimated based on the recognition results. The excitement level is quantified and estimated.

[0096] Once the excitement levels of all the people (audience) have been estimated (calculated), map data of the excitement levels is then created (step S24). The map data is data that represents the excitement level of each spectator on a map of the venue according to the position of each spectator in the captured image.

[0097] Once the map data has been created, danger areas are then estimated based on the created map data (step S25). The danger areas are estimated in the following procedure. First, the number of spectators whose excitement level is equal to or greater than a first threshold is counted for each partial area. The partial areas are set by dividing the spectator area into a plurality of areas. Next, partial areas in which the number of spectators whose excitement level is equal to or greater than the first threshold is equal to or greater than a second threshold are extracted. The extracted partial areas are determined to be danger areas.

[0098] When the risk area estimation process is completed, the movement route of the spectators is set based on the estimation result (step S26). The movement route is set for each partial area. Different movement routes are set depending on whether or not there is a risk area. If there is no risk area in the spectator area, the shortest route from each partial area to the nearest connecting passage is set as the movement route for each partial area. On the other hand, if there is a risk area in the spectator area, the movement route for the non-risk areas (partial areas other than the risk areas) is set, avoiding the risk area. Specifically, the movement route is set as follows. That is, first, the nearest connecting passage is identified from each partial area BA1 to BA12. The shortest route to the identified connecting passage is set (provisionally set) as the movement route for each partial area BA1 to BA12. Next, the partial areas (non-risk areas) that use the same connecting passage as the risk area are extracted. Next, the movement route for the partial areas (non-risk areas) other than the risk area that use the same connecting passage as the risk area is changed (reset). That is, a route that uses a connecting passage different from that of the spectators in the risk area is set. Specifically, the system then searches for the nearest connecting passage and sets the shortest route to the found connecting passage as the travel route. However, if a suitable connecting passage cannot be found within a predetermined distance, the system may set the shortest route to the nearest connecting passage as the travel route.

[0099] When the setting of the travel route is completed for each partial area (danger area and non-danger area), information on the set travel route is transmitted to the guidance display device 200 (step S27).

[0100] The process performed by the information processing device 100, i.e., the process of setting a travel route from images, may be configured to be performed continuously or at predetermined time intervals (at intervals). When performed continuously, the image capturing device 10 captures images of the spectator area as moving images, for example. The information processing device 100 processes the images input as moving images on a frame-by-frame basis to set a travel route. When processing at predetermined time intervals, the information processing device 100 may also be configured to input and process the moving images. In this case, the moving images are processed at predetermined frame intervals to set a travel route.

[0101] [Processing on the information display device] The guidance display device 200 receives information on the travel route transmitted from the information processing device 100 (step S31). The guidance display device 200 generates guidance information based on the received information on the travel route (step S32). The guidance display device 200 displays the generated guidance information on the display 210 (step S33).

[0102] When evacuating, leaving, or the like, spectators move around the venue following the guidance information displayed on the display 210 of the guidance display device 200.

[0103] As described above, according to the guidance system 1 of this embodiment, spectators in danger areas and spectators in non-danger areas can be guided separately. This allows spectators to be guided appropriately according to the situation at the venue. In particular, spectators in non-danger areas can be guided along a highly safe route. Furthermore, since the reason for the separate routes is kept secret, there is no risk of spectators in danger areas feeling resentful.

[0104] In addition, the guidance system 1 of this embodiment creates map data that shows the state (excitement level) of each spectator on a map according to the position of each spectator, so the situation in the venue can be accurately grasped. Also, the state of the spectators is quantified and shown as the excitement level, so dangerous areas can be automatically extracted.

[0105] [Second embodiment] In the guidance system of the present embodiment, the information processing device records information on the excitement level of each person (spectator) over time. In addition, when creating map data in the information processing device, the map data is created based on the history of the excitement level of each person.

[0106] The configuration other than the information processing device is the same as that of the guidance system of the first embodiment, so only the configuration of the information processing device will be described below.

[0107] [Configuration of information processing device] FIG. 23 is a block diagram of functions realized by the information processing device of this embodiment.

[0108] As shown in the figure, the information processing device 100 of this embodiment differs from the information processing device of the first embodiment in that it further includes an excitement level recording unit 124 and a maximum value detection unit 126. Therefore, only the differences from the information processing device of the first embodiment will be described below.

[0109] The functions of the excitement level recording section 124 and the maximum value detecting section 126 are realized by the CPU of the computer constituting the information processing device 100 executing a predetermined program.

[0110] The excitement level recording unit 124 records, over time, the excitement level of each audience member estimated by the estimation unit 114. In this embodiment, the excitement level is recorded in the HDD 104.

[0111] The maximum value detection unit 126 detects the maximum value of the excitement level of each audience member from the history of the excitement level of each audience member recorded over time in the HDD 104 (the maximum value in the history up to the time of detection).

[0112] The map data generating section 116 generates map data using the maximum value of the excitement level detected by the maximum value detecting section 126 .

[0113] [Operation of information processing device] The information processing device 100 of this embodiment performs a process of setting a movement route at a predetermined timing, specifically, when guiding evacuation or when all spectators are to exit at once.

[0114] FIG. 24 is a flowchart showing the procedure of the process performed in the information processing device.

[0115] First, a captured image is input (step S41). The captured image to be input is a moving image (including still images captured continuously at a fixed time interval). Next, people (spectators) are detected from the input image (one frame image) (step S42). Next, the excitement level of each detected person (spectator) is estimated (step S43). Next, the estimated excitement level of each person (spectator) is recorded in the HDD 104 (step S44). Next, it is determined whether or not a moving route needs to be set (step S45). That is, it is determined whether or not it is time to set a moving route. If a moving route is not set, the process returns to step S41, and the image of the next frame is processed. On the other hand, if a moving route is set, the maximum excitement level of each spectator is detected from the history of the excitement level of each spectator recorded in the HDD 104 (step S46). Next, map data is created using the information on the maximum excitement level of each detected spectator (step S47). Next, a dangerous area is estimated based on the created map data (step S48). Next, a travel route is set based on the information of the estimated dangerous area (step S49). Next, information of the set travel route is transmitted to the guidance display device 200 (step S50).

[0116] In this manner, the information processing device 100 of the present embodiment records the history of the excitement level of each spectator and creates map data based on the recorded history, thereby making it possible to extract potential dangerous areas from the past history.

[0117] In this embodiment, the maximum value is obtained from the history to create the map data, but the use of the history is not limited to this.Other than this, for example, the average excitement level of each audience member may be obtained from the history of the excitement level of each audience member, and the map data may be created from the obtained average excitement level.Furthermore, the number of times the excitement level was equal to or greater than the threshold value may be obtained, and the map data may be created from the obtained number of times.

[0118] [Variations] [Variations of the method for estimating a person's state] In the above embodiment, the state of the person is specified by the degree of excitement, but the method of specifying the state of the person is not limited to this. For example, the state of the person can be specified by the degree of a particular emotion (degree of anger, degree of joy, degree of fear, etc.).

[0119] In addition, when estimating a person's state, in the above embodiment, the facial expression, facial direction, and body movement of the person are recognized, and the state of the person is estimated based on the recognition result, but it is also possible to use any one of the information to estimate. For example, it is also possible to recognize only the facial expression of the person, and estimate the state of the person using only the recognition result of the facial expression. Alternatively, it is also possible to recognize only the body movement of the person, and estimate the state of the person using only the recognition result of the body movement.

[0120] In the above embodiment, when recognizing a facial expression, one of a plurality of predetermined types of facial expressions is specified as the most plausible one, but the method of recognizing a facial expression is not limited to this. For example, a method of classifying facial expressions into a plurality of types (for example, "happiness", "anger", "disgust", "surprise", "fear", "sadness", and "serious face (expressionless)", etc.), and determining the degree of each facial expression (likeliness of an expression), etc., can be adopted.

[0121] [Variations of the criteria] In the above embodiment, when estimating a person's state (excitement level), a score is given according to a predetermined criterion. The criterion itself may be set relatively. For example, in the case of a criterion for facial expression, the criterion may be determined from the cumulative value of each facial expression of the entire recognized face. This allows the criterion to be actively set according to the type of event, etc.

[0122] [Variations in the division of the spectator area] It is preferable that the spectator area be divided (divided into partial areas) appropriately, taking into consideration the size of the venue, the arrangement of seats, the layout of aisles within the venue, the number of entrances and exits, and the like.

[0123] [Map data output] The map data may be configured to be visualized and output. In this case, the information processing device is further provided with a function of a map data visualization processing unit. The visualized map data (so-called heat map, see FIG. 10) may be configured to be displayed on the display unit 106 of the information processing device 100, or may be further configured to be displayed on another display device (another example of the display unit). Furthermore, it may be configured to be printed out by a printer (an example of the output unit).

[0124] Like the map data, the information on the travel route can be configured to be displayed on the display unit 106 of the information processing device 100 or printed out from a printer.

[0125] In this way, by outputting visualized map data and information on travel routes, it is possible to provide convenience to people (such as security guards) when guiding spectators.

[0126] [Variations of the method for setting a travel route] The method of setting the movement route is not limited to the method described in the above embodiment. For example, a method of setting the movement route of each partial area from the information of the dangerous area using a trained model generated by machine learning, deep learning, etc. may be adopted.

[0127] [Another example of how route information is presented] In the above embodiment, the guidance display device 200 installed in the venue is configured to present information on a travel route and guidance information based on the travel route information, but the method of presenting information on the travel route, etc. is not limited to this.

[0128] 25 is a diagram showing another example of a method for presenting guide information. In the example shown in the figure, a projector 400 is used to present guide information to visitors. The projector 400 presents the guide information by projecting an arrow indicating the direction of travel onto a passageway.

[0129] Fig. 26 is a diagram showing another example of a method for presenting guidance information. This diagram shows an example of presenting guidance information by physical display. In the example shown in this diagram, a movable signboard 500 is used to present the guidance information. The signboard 500 shown in this diagram is a movable signboard equipped with casters, and displays an arrow indicating the direction of travel to present the guidance information.

[0130] By using these techniques, it is possible to present route information to spectators without requiring them to carry special devices (such as mobile terminals).

[0131] In addition, a configuration may be adopted in which spectators are provided with special devices (for example, mobile terminals such as smartphones and tablet terminals, wearable computers such as smart glasses and smart watches, and personal computers, etc.) to present information on the movement route and the like. For example, the information on the movement route can be transmitted to the mobile terminals and wearable computers owned by the spectators, and the information on the movement route can be presented as characters and images on the screens, etc. In this case, information on the position of each spectator (for example, information on the seat position) is managed, and information on the movement route according to the position of each spectator is transmitted. In other words, information on the movement route of a partial area where the spectator is present is transmitted. Information on the position of the spectators is managed in association with information on the mobile terminals owned by the spectators. This association is performed, for example, when purchasing tickets. Alternatively, it is performed afterwards by each spectator registering information on his or her own mobile terminal (telephone number, email address, etc.). Management is performed, for example, by a server. The server receives information on the movement route from an information processing device, generates guide information, etc., and transmits it to the mobile terminals of each spectator.

[0132] [Modification of the configuration of the information processing device] Some or all of the functions of the information processing device can be realized by various processors. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes a program and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electric circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to execute a specific process. A program is synonymous with software.

[0133] A processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types. For example, a processing unit may be configured with multiple FPGAs, or a combination of a CPU and an FPGA. Also, multiple processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, there is a form in which a processor that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip is used, as represented by a system on chip (SoC). In this way, various processing units are configured using one or more of the above various processors as a hardware structure. [Explanation of symbols]

[0134] 1 Guidance System 10. Imaging Device 12 Camera 100 Information processing device 101 CPU 103 ROM 104 HDD 105 Operation section 106 Display section 107 Image input section 108 Communications Department 110 Image acquisition unit 112 Person detection unit 114 Estimation Department 114A Facial expression recognition unit 114B Orientation recognition unit 114C Motion recognition unit 114D Scoring Department 114E Excitement level calculation section 116 Map Data Creation Department 118 Danger Area Estimation Unit 118A Counting Unit 118B Danger area extraction section 120 Movement route setting section 122 Transmitter 124 Excitement Recording Unit 126 Maximum value detector 200 Guidance display device 210 Display 220 Display control unit 220A Travel route information receiving unit 220B Guidance information generation unit 220C Display control unit 300 Event Venues 310 Ground 320 Spectator Area 322 1st floor seating area 324 2nd floor seating area 326 Passage (Circular Passage) 328 Passageway (Inter-block Passageway) 328A Handrail 330 Outer Area 332 Passageway (connecting passageway) 332A~332C connecting passage 332X Entrance to the connecting passage Gate 334A Gate 334B Gate 334C Gate 334D 400 Projector 500 Signboard BA Partial Area BA1~BA12 Partial Area IP People Images Part of the image taken by IS1 In Guide Information Image In1 Area Map In2 Arrow indicating direction of travel Image of MI map R1 Movement Path W Shooting range dp dot S11~S12 Processing procedure at the imaging device S21 to S27: Processing procedure in information processing device S31~S33 Processing procedure for the information display device S41 to S50: Processing procedure in information processing device

Claims

1. A processor is provided. The processor, Dividing the area to be processed into a number of subareas; extracting a first area from the plurality of partial areas based on information obtained by quantifying a state of a person and information on a position of the person; setting a movement route of a person in the first area and / or a person in a second area other than the first area; It is configured as follows: Information processing device.

2. The processor is configured to count the number of the persons whose status is equal to or greater than a first threshold for each of the partial areas, and to extract the partial areas in which the number of the persons whose status is equal to or greater than the first threshold is equal to or greater than a second threshold as the first areas. The information processing device according to claim 1 .

3. The processor further comprises: Obtaining an image of the area; Detecting a person from the image; A state of the detected person is estimated by quantifying the state of the person; acquiring numerical information on the state of the person and information on the position of the person within the image; It is configured as follows: The information processing device according to claim 1 .

4. The processor further comprises: Transmitting information about the set travel route. It is configured as follows: The information processing device according to claim 1 .

5. The processor, Recognizing at least one of a facial expression and a body movement of the person from the detected image of the person; quantifies and estimates the state of the person based on the result of the recognition; It is configured as follows: The information processing device according to claim 3 .

6. The processor, giving a score to the result of the recognition according to a predetermined criterion, and quantifying the state of the person; It is configured as follows: The information processing device according to claim 5 .

7. The processor, Recognizing the facial expression, facial direction, and body movement of the person from the detected image of the person; giving scores to the results of the recognition of the facial expression, facial direction, and body movement of the person according to a predetermined standard, and calculating the sum of the scores to quantify the state of the person; It is configured as follows: The information processing device according to claim 3 .

8. The processor, recording the estimated state of the person over time; detecting a maximum value of the state of the person recorded over time; creating data in which numerical information on the state of the person is associated with information on the position of the person on a map using the maximum value; It is configured as follows:

8. The information processing device according to claim 3, 5, 6 or 7.

9. The image is an image captured of a venue having a seating area in which a plurality of seats are arranged as the area, The first area and the second area are defined within the seating area. The information processing device according to claim 8.

10. The processor further comprises: outputting visualized data in which the information on the state of the person is associated with information on the position of the person on a map, and / or outputting information on the path of movement; It is configured as follows: The information processing device according to claim 1 .

11. The present invention further comprises a display configured to display visualized data in which the quantified information on the state of the person and the information on the position of the person on a map are associated with each other, and / or information on the movement path. The information processing device according to claim 1 .

12. A processor is provided. The processor, Obtain an image of the area to be processed; Detecting a person from the image; A state of the detected person is estimated by quantifying the state of the person; Dividing the area into a plurality of sub-areas; extracting a first area from the plurality of partial areas based on information obtained by quantifying a state of a person and information on a position of the person; setting a movement route of a person in the first area and / or a person in a second area other than the first area; Recording quantified information on the state of the person over time; It is configured as follows: Information processing device.

Images