Information presentation system, information presentation method, computer-readable recording medium, and distress degree determination system

Abstract

The present invention provides an information presentation system, an information presentation method, a computer-readable recording medium, and a degree of trouble determination system. An action analysis section (110) analyzes monitoring data obtained by monitoring a monitoring site, and detects a corresponding action that is performed at the monitoring site and has a feature in common with a violation action. A situation analysis section (120) analyzes image data obtained by imaging the monitoring site, and detects a surrounding situation of an action position corresponding to a position at which the corresponding action is performed. The situation analysis section determines a degree of trouble of the corresponding action based on the surrounding situation. A presentation control section (130) determines a warning level based on the degree of trouble. The presentation control section presents presentation information corresponding to the warning level using a presentation device provided at the monitoring site.

Description

Technical Field

[0001] The present disclosure relates to information prompts for inhibiting disturbing behaviors. Background Art

[0002] Disturbing behavior such as impoliteness in public places causes discomfort to those around them. Furthermore, being reminded of impoliteness by others can also cause discomfort to the offender, potentially leading to trouble.

[0003] Patent Document 1 discloses a technique for detecting disturbing behavior of passengers in a passenger compartment and requesting the perpetrator of the disturbing behavior to stop the disturbing behavior.

[0004] Specifically, in the technology of Patent Document 1, individual notification devices are associated with each passenger in the passenger compartment. These devices are installed on a portable terminal or seat and are capable of providing individual notifications to each passenger in the passenger compartment. Furthermore, in the technology of Patent Document 1, the individual causing trouble in the passenger compartment is identified through methods such as detection of images within the passenger compartment or notifications from other passengers. The individual notification device associated with the individual causing trouble then notifies the individual of a request to stop the disturbing behavior.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2020-3935 Summary of the Invention

[0008] Problems to be solved by the invention

[0009] The technology of Patent Document 1 is based on the premise that each passenger can be identified in a closed, narrow space and on the premise that an individual notification device can be provided to each passenger individually.

[0010] Therefore, the technology of Patent Document 1 cannot be used in a public space used by an unspecified number of people.

[0011] Furthermore, if the violator is strongly reminded when there are people around, the violator will feel uncomfortable and may get into trouble.

[0012] The purpose of the present disclosure is to be able to remind people of disturbing behavior while avoiding trouble in public spaces.

[0013] Means used to solve problems

[0014] The information prompting system disclosed herein comprises: an action detection unit that analyzes surveillance data obtained by monitoring a surveillance location and detects corresponding actions conducted at the surveillance location and having characteristics common to the violation actions; a situation detection unit that analyzes image data obtained by photographing the surveillance location and detects surrounding conditions of an action location corresponding to the location where the corresponding action was conducted; a disturbance level determination unit that determines the disturbance level of the corresponding action based on the surrounding conditions; a level determination unit that determines a reminder level based on the disturbance level; and an information prompting unit that uses a prompting device installed at the surveillance location to prompt prompt information corresponding to the reminder level.

[0015] Effects of the Invention

[0016] According to the present disclosure, it is possible to warn people about disturbing behaviors at a level appropriate to the surrounding situation, thereby preventing trouble in a public space. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a structural diagram of the information presentation system 200 in the first embodiment.

[0018] Figure 2 This is a functional configuration diagram of the information presentation device 100 in the first embodiment.

[0019] Figure 3 This is a structural diagram of the monitoring location 201 in the first embodiment.

[0020] Figure 4 This is a flowchart of the information presentation method in the first embodiment.

[0021] Figure 5 This is a flowchart of the behavior analysis ( S110 ) in the first embodiment.

[0022] Figure 6 This is a diagram showing the feature data group 191 in the first embodiment.

[0023] Figure 7 This is a diagram showing the violation possibility data 192 in the first embodiment.

[0024] Figure 8 This is a flowchart of the situation analysis ( S120 ) in the first embodiment.

[0025] Figure 9 This is a diagram showing the ambient condition data group 193 in the first embodiment.

[0026] Figure 10 This is a diagram showing the facial expression distress level data 194A in the first embodiment.

[0027] Figure 11This is a diagram showing behavioral trouble level data 194B in the first embodiment.

[0028] Figure 12 This is a diagram showing notification trouble level data 194C in the first embodiment.

[0029] Figure 13 This is a diagram showing the comprehensive disturbance level data 194D in the first embodiment.

[0030] Figure 14 This is a flowchart of the presentation control ( S130 ) in the first embodiment.

[0031] Figure 15 This is a diagram showing the final judgment possibility data 195 in the first embodiment.

[0032] Figure 16 This is a diagram showing the warning level data 196 in the first embodiment.

[0033] Figure 17 This is a flowchart of step S132 in the first embodiment.

[0034] Figure 18 This is a diagram showing the reminder information data 197 in the first embodiment.

[0035] Figure 19 This is a flowchart of the information prompting method in implementation mode 3.

[0036] Figure 20 This is a flowchart of the behavior analysis (S310) in the third embodiment.

[0037] Figure 21 This is a flowchart of the information prompting method in implementation mode 4.

[0038] Figure 22 This is a flowchart of the behavior analysis (S410) in the fourth embodiment.

[0039] Figure 23 This is a flowchart of the information prompting method in implementation mode 5.

[0040] Figure 24 This is a flowchart of the presentation control (S530) in the fifth embodiment.

[0041] Figure 25 This is a flowchart of step S532 in the fifth embodiment.

[0042] Figure 26 This is a diagram showing the warning level data 198A in the fifth embodiment.

[0043] Figure 27 This is a diagram showing the warning level data 198B in the fifth embodiment.

[0044] Figure 28 This is a diagram showing the warning level data 198C in the fifth embodiment.

[0045] Figure 29 2 is a hardware configuration diagram of the information presentation device 100 according to the embodiment. DETAILED DESCRIPTION

[0046] In the embodiments and the accompanying drawings, the same reference numerals are used for the same or corresponding elements. The description of elements marked with the same reference numerals as the elements described is omitted or simplified as appropriate. The arrows in the figures mainly indicate data flows or processing procedures.

[0047] Implementation method 1.

[0048] based on Figures 1 to 18 The information presentation system 200 will be described.

[0049] ***Structure description***

[0050] based on Figure 1 The configuration of the information presentation system 200 will be described.

[0051] The information presentation system 200 includes the information presentation device 100 .

[0052] The information presentation device 100 is a computer including hardware such as a processor 101, a main storage device 102, an auxiliary storage device 103, an input interface 104, an output interface 105, and a communication device 106. These hardware components are connected to each other via signal lines.

[0053] The processor 101 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 101 is a CPU, a DSP, or a GPU.

[0054] IC is the abbreviation of Integrated Circuit.

[0055] CPU is the abbreviation of Central Processing Unit.

[0056] DSP is the abbreviation of Digital Signal Processor.

[0057] GPU is the abbreviation of Graphics Processing Unit.

[0058] The main storage device 102 is a volatile or non-volatile storage device. The main storage device 102 is also called a memory or a main memory. For example, the main storage device 102 is a RAM. Data stored in the main storage device 102 is saved in the auxiliary storage device 103 as necessary.

[0059] RAM is an abbreviation for Random Access Memory.

[0060] The auxiliary storage device 103 is a non-volatile storage device. For example, the auxiliary storage device 103 is a ROM, an HDD, or a flash memory. Data stored in the auxiliary storage device 103 is loaded into the main storage device 102 as necessary.

[0061] ROM is an abbreviation for Read Only Memory.

[0062] HDD is an abbreviation for Hard Disk Drive.

[0063] The input interface 104 is a port connected to an input device and a monitoring device 210. For example, the input device is a keyboard and a mouse used for operating the information presentation device 100. For example, the monitoring device 210 is a camera 211 and a microphone 212. Input to the information presentation device 100 is performed using the input interface 104.

[0064] The output interface 105 is a port connected to an output device and an output device 220. For example, the output device is a display for an operator who operates the information presentation device 100. For example, the output device 220 is a display 221, a speaker 222, a projector 223, and an illuminating device 224. Output from the information presentation device 100 is performed using the output interface 105.

[0065] The communication device 106 is a receiver and a transmitter. For example, the communication device 106 is a communication chip or a NIC. Communication of the information presentation device 100 is performed using the communication device 106. For example, the information presentation device 100 communicates with a portable terminal 231 using the communication device 106.

[0066] NIC is an abbreviation for Network Interface Card.

[0067] The information presentation device 100 has elements such as a movement analysis section 110, a situation analysis section 120, and a presentation control section 130. These elements are implemented by software.

[0068] The auxiliary storage device 103 stores an information presentation program for causing the computer to function as the behavior analysis unit 110, the situation analysis unit 120, and the presentation control unit 130. The information presentation program is loaded into the main storage device 102 and executed by the processor 101.

[0069] The auxiliary storage device 103 also stores an OS. At least a portion of the OS is loaded into the main storage device 102 and executed by the processor 101.

[0070] The processor 101 executes the information presentation program while executing the OS.

[0071] OS is the abbreviation of Operating System.

[0072] Input and output data of the information presentation program are stored in the storage unit 190 .

[0073] The main storage device 102 functions as the storage unit 190 . However, storage devices such as the auxiliary storage device 103 , registers within the processor 101 , and cache memory within the processor 101 may function as the storage unit 190 instead of or in addition to the main storage device 102 .

[0074] The information presentation device 100 may include a plurality of processors instead of the processor 101 .

[0075] The information presentation program can be recorded (stored) in a computer-readable form on a non-volatile recording medium such as an optical disk or a flash memory.

[0076] based on Figure 2 The functional structure of the information presentation device 100 will be described.

[0077] The behavior analysis unit 110 includes a behavior detection unit 111 and a behavior determination unit 112 .

[0078] The monitoring data is input to the behavior analysis unit 110 .

[0079] The monitoring data is data obtained by the monitoring device 210 .

[0080] The monitoring data obtained by the camera 211 is referred to as “video data”.

[0081] The monitoring data obtained by the microphone 212 is referred to as "sound data".

[0082] The situation analysis unit 120 includes a situation detection unit 121 , a notification reception unit 122 , and a trouble level determination unit 123 .

[0083] The notification data is input to the situation analysis unit 120 .

[0084] The notification data is data indicating information notified from the portable terminal 231 .

[0085] The presentation control unit 130 includes a level determination unit 131 , an information selection unit 132 , a device selection unit 133 , and an information presentation unit 134 .

[0086] The presentation data is output from the presentation control unit 130 .

[0087] The presentation data is data indicating information presented using the output device 220 .

[0088] based on Figure 3 The structure of the monitoring location 201 will be described.

[0089] The monitoring place 201 is a place monitored by the monitoring device 210. In particular, the monitoring place 201 is a place used by an unspecified number of people. For example, the monitoring place 201 is a public place (space).

[0090] A person who is present at the monitored location 201, that is, a person who uses the monitored location 201 is referred to as a "user 202".

[0091] Disturbing behavior, impolite behavior, and other actions prohibited in the monitored location 201 are referred to as "violations."

[0092] Actions that share characteristics with illegal actions, that is, actions that may cause trouble to others, are called "corresponding actions."

[0093] A position corresponding to the position where the corresponding action is performed is referred to as an "action position."

[0094] The user 202 who performs the corresponding action is referred to as an “actor 203 ”.

[0095] The users 202 located around the actor 203 are referred to as “surrounding persons 204 ”.

[0096] Monitoring equipment 210 such as a camera 211 and a microphone 212 is installed in the monitoring location 201 .

[0097] The monitoring device 210 is a device for monitoring the monitoring site 201 .

[0098] The camera 211 captures an image of the monitored location 201 and outputs image data.

[0099] The microphone 212 collects sound at the monitoring location 201 and outputs sound data.

[0100] A plurality of cameras 211 may be installed at different locations in the monitoring location 201. A plurality of microphones 212 may also be installed at different locations. In addition, one or more monitoring devices 210 other than the cameras 211 and the microphones 212 may also be installed.

[0101] The position information and the like of each monitoring device 210 are stored in advance in the storage unit 190 .

[0102] Various output devices 220 such as a display 221 , a speaker 222 , a projector 223 , and a lighting device 224 are installed in the monitoring location 201 .

[0103] The output device 220 is a device for presenting reminder information for the actor 203 to refrain from a corresponding action.

[0104] A plurality of various output devices 220 may be installed at different locations in the monitoring location 201. Furthermore, various output devices 220 or one or more output devices 220 of different types may be installed.

[0105] Position information and the like of each output device 220 are stored in advance in the storage unit 190 .

[0106] ***Action description***

[0107] The operation steps of the information presentation system 200 correspond to the information presentation method. In addition, the operation steps of the information presentation device 100 correspond to the processing steps performed based on the information presentation program.

[0108] based on Figure 4 The information prompt method is explained.

[0109] In step S110 , the behavior analysis unit 110 performs behavior analysis.

[0110] In the behavior analysis, the behavior detection unit 111 analyzes monitoring data obtained by monitoring the monitoring location 201 and detects corresponding behavior.

[0111] The corresponding action is an action performed at the monitoring location 201 and having characteristics common to the violation action.

[0112] Step S110 will be described in detail later.

[0113] In step S120 , the situation analysis unit 120 performs situation analysis.

[0114] In the situation analysis, the situation detection unit 121 analyzes the image data obtained by photographing the monitoring location 201 to detect the surrounding situation of the action position.

[0115] Then, the trouble level determination unit 123 determines the trouble level of the corresponding action based on the surrounding conditions.

[0116] Step S120 will be described in detail later.

[0117] In step S130 , the presentation control unit 130 performs presentation control.

[0118] In the presentation control, the level determination unit 131 determines the presentation level based on the degree of distress.

[0119] Then, the information presentation unit 134 presents presentation information corresponding to the warning level using a presentation device installed in the monitoring location 201 .

[0120] The presentation device is an output device 220 selected for presentation of information.

[0121] Step S130 will be described in detail later.

[0122] based on Figure 5 The behavior analysis ( S110 ) will be described in detail.

[0123] In step S111 , the action detection unit 111 analyzes monitoring data obtained by monitoring the monitoring location 201 and detects corresponding actions.

[0124] Step S111 will be described in detail.

[0125] First, the motion detection unit 111 receives video data.

[0126] Then, the action detection unit 111 detects the corresponding action by analyzing the image data, identifies the actor 203 , and calculates the position of the actor 203 .

[0127] The position of the actor 203 becomes the action position.

[0128] In addition, a plurality of violation actions are predefined, and the action detection unit 111 detects corresponding actions for the plurality of violation actions.

[0129] Step S111 will be described in detail. The action detection unit 111 analyzes the video data in sequence and performs the processes (1) to (5).

[0130] (1) The behavior detection unit 111 detects each user 202 appearing in the video represented by the video data.

[0131] (2) The behavior detection unit 111 extracts characteristic information of each user's 202 behavior from the image data. The characteristic information represents one or more characteristics that identify the behavior. For example, the characteristic information represents the position of each joint and changes in the position of each joint. The extracted characteristic information is referred to as "extracted information."

[0132] (3) The behavior detection unit 111 detects each user 202 from the feature data group 191 (refer to Figure 6 ) is searched for feature information that represents features common to the extracted information. The feature information to be searched is called "common information."

[0133] Figure 6 The characteristic data group 191 is shown. The characteristic data group 191 is stored in the storage unit 190 in advance.

[0134] The feature data group 191 includes feature data of each of one or more types of violations.

[0135] The characteristic data associates the type of violation with characteristic information of the violation.

[0136] (4) When common information is found, the user 202 is the actor 203 and the action of the user 202 is a corresponding action.

[0137] The action detection unit 111 extracts feature data representing common information from the feature data group 191. The type represented by the extracted feature data is the type of the violation action corresponding to the corresponding action, and corresponds to the type of the corresponding action.

[0138] (5) The action detection unit 111 analyzes the image data and calculates the position of the actor 203 .

[0139] For example, storage unit 190 pre-stores coordinate conversion data that represents the correspondence between the three-dimensional coordinate values ​​of various features located at monitoring location 201 and the two-dimensional coordinate values ​​of each feature within the image. The action detection unit 111 then calculates the three-dimensional coordinate values ​​of actor 203 based on the image data and the positional relationship between actor 203 and the various features within the image. For example, if actor 203 is located between a first feature and a second feature, action detection unit 111 uses the three-dimensional coordinate values ​​of the first and second features to calculate the three-dimensional coordinate values ​​of the point between the first and second features. The calculated three-dimensional coordinate values ​​are the three-dimensional coordinate values ​​of actor 203. The various features located at monitoring location 201 can be monitoring device 210, output device 220, or other objects.

[0140] In step S112 (refer to Figure 5 ), the action determination unit 112 calculates the degree of consistency between the corresponding action and the violating action based on the feature information of the corresponding action and the feature information of the violating action.

[0141] Step S112 will be described in detail.

[0142] The characteristic information of the corresponding action is obtained in step S111 (2).

[0143] The characteristic information of the violation action is obtained in step S111 (4).

[0144] The action determination unit 112 calculates the ratio of the common features between the feature information of the violation action and the feature information of the corresponding action. The calculated ratio is the degree of consistency between the corresponding action and the violation action.

[0145] For example, when the number of features indicated by the feature information of the violating action is "10" and the number of features common to the feature information of the violating action and the feature information of the corresponding action is "7", the degree of consistency is "70%".

[0146] “%” means percentage.

[0147] In step S113 , the action determination unit 112 determines the possibility of a violation action of the corresponding action based on the degree of consistency between the corresponding action and the violation action.

[0148] The violation action possibility corresponds to the consistency level indicating the degree of consistency, and indicates the possibility that the corresponding action is a violation action.

[0149] Specifically, the action determination unit 112 determines the violation action possibility data 192 (see Figure 7 ), select the violation action possibility corresponding to the consistency. The selected violation action possibility is the violation action possibility of the corresponding action.

[0150] exist Figure 7 , the violation action possibility data 192 for the violation action A is shown. The violation action possibility data 192 is stored in advance in the storage unit 190 for each type of violation action.

[0151] The violation action possibility data 192 associates the range of the degree of consistency with the violation action possibility for each range of the degree of consistency.

[0152] exist Figure 7 In the , three levels of consistency are defined. Level 1 is a high level of consistency, 70% or higher. Level 2 is a medium level of consistency, 30% or higher but less than 70%. Level 3 is a low level of consistency, less than 30%.

[0153] When the level of consistency between the corresponding action and the violating action A is level 1, the violating action possibility is “high”.

[0154] When the level of consistency between the corresponding action and the illegal action A is level 2, the possibility of illegal action is "doubtful".

[0155] When the level of consistency between the corresponding action and the violation action A is level 3, the violation action possibility is “low”.

[0156] based on Figure 8 The situation analysis ( S120 ) will be described in detail.

[0157] In step S121 , the situation detection unit 121 analyzes the image data to detect the surrounding situation of the action position.

[0158] Step S121 will be described in detail.

[0159] The situation detection unit 121 detects the presence or absence of the surrounding person 204 as one of the information indicating the surrounding situation.

[0160] When the surrounding person 204 exists, the situation detection unit 121 detects at least one of the expression of the surrounding person 204 and the behavior of the surrounding person 204 as one of the information indicating the surrounding situation.

[0161] Step S121 will be described in detail. The action determination unit 112 analyzes the image data in sequence and performs the processes (1) to (6).

[0162] (1) The situation detection unit 121 analyzes the video data and calculates the position of each user 202 (excluding the actor 203). The calculation method is the same as the method (5) of step S111.

[0163] (2) The situation detection unit 121 obtains the surrounding condition data set 193 (see Figure 9 ) extracts the surrounding conditions for the type of the corresponding action. The type of the corresponding action is the type of the violation action corresponding to the corresponding action, and is determined by (4) of step S111.

[0164] exist Figure 9 1 shows the surrounding condition data group 193. The surrounding condition data group 193 is stored in the storage unit 190 in advance.

[0165] The surrounding condition data group 193 includes surrounding condition data for each of one or more types of violation actions.

[0166] The surrounding condition data associates the type of the violation action with the surrounding conditions of the violation action.

[0167] The surrounding conditions are the range of trouble caused by the violation, that is, the range considered to be around the violation, and are defined according to the type of violation. For example, the surrounding conditions are defined by the distance from the action location as the base point. The position of the actor 203 corresponding to the action location is calculated by (5) of step S111.

[0168] The surrounding conditions of each violation action may be different from or the same as the surrounding conditions of other violation actions.

[0169] The shape of the range determined according to various surrounding conditions may be either circular or rectangular, or may be other shapes.

[0170] (3) The situation detection unit 121 identifies the users 202 that are present at each location that satisfies the extracted surrounding conditions. The identified users 202 are the surrounding persons 204 .

[0171] When the user 202 does not exist at each position satisfying the extracted surrounding conditions, the surrounding person 204 does not exist.

[0172] When the user 202 exists at a location that satisfies the extracted surrounding conditions, a surrounding person 204 exists.

[0173] (4) When the surrounding persons 204 are present, the situation detection unit 121 analyzes the video data and detects the expressions and emotions of each surrounding person 204 .

[0174] Specifically, the situation detection unit 121 recognizes features of the overall facial movement of the surrounding person 204, such as widening or lowering of the corners of the mouth. The situation detection unit 121 then determines an emotion such as anger, disgust, or dissatisfaction based on the recognized features.

[0175] For example, various machine learning methods, such as learning models using training data, are used to detect facial expressions and emotions. Alternatively, open source software can be used.

[0176] (5) When the surrounding persons 204 are present, the situation detection unit 121 analyzes the video data and detects the behavior of each surrounding person 204 .

[0177] Specifically, the situation detection unit 121 detects actions such as separation from the actor 203 , avoidance from the actor 203 , and watching the actor 203 .

[0178] For example, various machine learning methods, such as learning models using training data, are used to detect actions. Alternatively, open source software can be used.

[0179] (6) When there are surrounding persons 204 , the situation detection unit 121 counts the number of surrounding persons 204 .

[0180] In step S122 (refer to Figure 8 ), when there are surrounding persons 204, each surrounding person 204 uses the portable terminal 231 to send a trouble notification.

[0181] Then, the notification accepting unit 122 accepts trouble notifications transmitted from the mobile terminals 231 of the surrounding persons 204 within a certain period of time.

[0182] The trouble notification is a notification for reporting an action that causes trouble to the surrounding person 204, and shows information such as the notification time, notification location, and action type.

[0183] For example, the information presentation system 200 includes a web server. The web server manages a trouble notification website for each monitored location 201. The trouble notification website has a function (e.g., a graphical user interface) for receiving information such as the presence of a trouble, the location of the troubled person (e.g., the location of the portable terminal 231), and the type of action causing the trouble. Furthermore, the trouble notification website has a function for transmitting a trouble notification to the information presentation device 100, indicating the received information and the time of notification.

[0184] Then, the surrounding person 204 accesses the trouble notification website using the mobile terminal 231 to send a trouble notification.

[0185] In step S123 , the trouble level determination unit 123 determines the trouble level of the corresponding activity based on the surrounding conditions of the activity location and the number of trouble notifications.

[0186] Step S123 will be described in detail. The trouble level determination unit 123 performs the processes (1) to (4).

[0187] (1) The distress level determination unit 123 uses the distress level data 194A (see Figure 10 ) selects the distress level based on the expression of each surrounding person 204, and determines the distress level based on the expression based on the selected distress level. For example, the distress level determination unit 123 determines that the highest distress level among the selected distress levels is the distress level based on the expression.

[0188] exist Figure 10 2 shows the expression distress level data 194A. The expression distress level data 194A is stored in the storage unit 190 in advance.

[0189] The expression distress level data 194A associates the type of expression and the distress level based on the expression with each other for each type of expression.

[0190] For example, the distress level based on an angry expression, a disgusted expression, or a dissatisfied expression is “high.” Also, the distress level based on an expression expressing other emotions is “low.”

[0191] The expression distress level data 194A may be prepared for each monitoring location 201. The types of expressions and the relationship between the types and distress levels may be appropriately defined based on the conditions of the monitoring location 201 such as the width and the number of users.

[0192] (2) The disturbance level determination unit 123 follows the action disturbance level data 194B (see Figure 11 ) selects the distress level based on the behavior of each surrounding person 204, and determines the distress level based on the behavior based on the selected distress level. For example, the distress level determination unit 123 determines that the highest distress level among the selected distress levels is the distress level based on the behavior.

[0193] exist Figure 11 , the behavioral trouble level data 194B is shown. The behavioral trouble level data 194B is stored in the storage unit 190 in advance.

[0194] The behavior trouble level data 194B associates the behavior type and the trouble level based on the behavior with each other for each behavior type.

[0195] For example, the degree of annoyance based on actions such as separating from the actor 203, avoiding the actor 203, or watching the actor 203 is “large.” In addition, the degree of annoyance based on other actions is “small.”

[0196] The behavioral annoyance level data 194B may be prepared for each monitoring location 201. The types of behaviors and the relationship between the types and the annoyance levels may be appropriately defined based on the conditions of the monitoring location 201 such as the width and the number of users.

[0197] (3) First, the trouble level determination unit 123 determines the trouble level notification for the corresponding action among the received trouble level notifications.

[0198] Specifically, the trouble level determination unit 123 compares the notification time, notification location, and action type indicated by each trouble notification with the detection time, action location, and action type of the corresponding action, thereby identifying the trouble notification for the corresponding action.

[0199] Next, the trouble level determination unit 123 counts the number of trouble notifications for the corresponding action.

[0200] Then, the trouble level determination unit 123 uses the notification trouble level data 194C (see Figure 12 ) selects a trouble level corresponding to the number of trouble notifications for the corresponding action. The selected trouble level is based on the trouble level of the notification.

[0201] exist Figure 12 194C of the trouble level data for notification is shown in FIG. The trouble level data for notification 194C is stored in the storage unit 190 in advance.

[0202] The notification trouble level data 194C associates the number of notifications and the trouble level based on the notifications with each other for each number of notifications.

[0203] For example, when the number of notifications is “x” or greater, the annoyance level due to the notifications is “high.” On the other hand, when the number of notifications is less than “x,” the annoyance level due to the notifications is “low.”

[0204] Notification distress level data 194C may also be prepared for each monitored location 201. The threshold "x" for the number of notifications can be appropriately defined based on the conditions of the monitored location 201, such as its size and the number of users. For example, a threshold of "1" may be defined for a monitored location 201 where only a certain number of people can enter. In this case, if there is also one distress notification, the distress level is considered "high."

[0205] (4) The disturbance level determination unit 123 uses the comprehensive disturbance level data 194D (refer to Figure 13 ), a comprehensive distress level corresponding to the distress level based on expression, the distress level based on action, and the distress level based on notification is selected. The selected comprehensive distress level becomes the distress level of the corresponding action.

[0206] exist Figure 13 The comprehensive disturbance level data 194D is shown in FIG. The comprehensive disturbance level data 194D is stored in the storage unit 190 in advance.

[0207] The comprehensive distress level data 194D shows the comprehensive distress level for each combination of the distress level based on expression, the distress level based on action, and the distress level based on notification.

[0208] However, the troublesomeness determination unit 123 may determine any one of the troublesomeness based on expression, the troublesomeness based on action, and the troublesomeness based on notification as the troublesomeness based on the corresponding action.

[0209] Alternatively, the distress level determination unit 123 may determine a comprehensive distress level obtained by combining any two of the distress levels of expression, action, and notification as the distress level of the corresponding action.

[0210] based on Figure 14 The prompt control ( S130 ) will be described in detail.

[0211] In step S131, the device selection unit 133 selects an output device 220 to be used as a prompting device from among the plurality of output devices 220 based on at least one of the consistency level of the corresponding action with respect to the violating action, the action location of the corresponding action, and the surrounding conditions of the corresponding action. The device selection unit 133 may also select two or more prompting devices.

[0212] Step S131 will be described in detail. The possibility of a violation action corresponds to the consistency level.

[0213] The device selection unit 133 selects a presentation device according to a selection rule. The selection rule for the presentation device is defined in advance.

[0214] For example, when the violation possibility is "low", the device selection unit 133 selects the speaker 222 as the presentation device. However, the device selection unit 133 may select the display 221, the speaker 222, the projector 223, or a combination thereof as the presentation device.

[0215] For example, if the likelihood of a violation is "suspicious" or "high," device selection unit 133 selects a notification device based on the location of the action and surrounding conditions. If the number of people around 204 exceeds a threshold, device selection unit 133 selects speaker 222 as the notification device. Furthermore, regardless of the number of people around 204, device selection unit 133 selects speaker 222 or projector 223 closest to the location of the action as the notification device.

[0216] In step S132 , the level determination unit 131 determines the warning level based on the degree of bothersomeness of the corresponding action, the consistency level of the corresponding action with respect to the violating action, and the presence or absence of the surrounding person 204 .

[0217] The reminder level is the level of reminder for the actor 203 .

[0218] Step S132 will be described in detail. The level determination unit 131 performs the processes (1) and (2). The likelihood of a violation based on the degree of consistency corresponds to the degree of consistency.

[0219] (1) The level determination unit 131 determines the final determination possibility based on the degree of bother of the corresponding action and the possibility of the violation action based on the degree of consistency.

[0220] The trouble level of the corresponding action is obtained through step S123.

[0221] The possibility of a violation action based on the degree of consistency is obtained in step S113 .

[0222] The final judgment probability is the final judgment probability of the violation action.

[0223] Specifically, the level determination unit 131 uses the final judgment probability data 195 (see Figure 15 ), select the final judgment possibility corresponding to the possibility of violation action based on consistency and the degree of trouble of the corresponding action.

[0224] exist Figure 15 2 shows the final judgment possibility data 195. The final judgment possibility data 195 is stored in the storage unit 190 in advance.

[0225] The final determination possibility data 195 shows the final determination possibility for each combination of the degree of conformity-based deviant behavior possibility and the degree of disturbance of the corresponding action.

[0226] For example, in a case where the degree of conformity-based deviant behavior possibility is "high" and the degree of disturbance of the corresponding action is "large", the final determination possibility is determined to be "high".

[0227] Further, in a case where the degree of conformity-based deviant behavior possibility is "suspicious" and the degree of disturbance of the corresponding action is "small", the final determination possibility is determined to be "suspicious". In this case, the corresponding action can develop into deviant behavior, and it is expected that the occurrence of the deviant behavior can be prevented by prompting the alert information.

[0228] (2) The level determination section 131 determines the alert level based on the final determination possibility and the presence or absence of the surrounding person 204.

[0229] The presence or absence of the surrounding person 204 is detected in step S121 as one of the information indicating the surrounding situation.

[0230] Specifically, the level determination section 131 selects the alert level corresponding to the final determination possibility and the presence or absence of the surrounding person 204 from the alert level data 196 (refer to Fig. 10). Figure 16 ).

[0231] The alert level data 196 is shown in Fig. 10. The alert level data 196 is stored in the storage section 190 in advance. Figure 16

[0232] The alert level data 196 shows the alert level for each combination of the final determination possibility and the presence or absence of the surrounding person 204.

[0233] For example, the alert level is defined in four stages. The higher the level, the stronger the intensity of the alert (warning).

[0234] In the alert level 1 "general enlightenment", general enlightenment for notifying the etiquette of the place and the rules of the place is performed. The actor 203 is not particularly alerted and does not feel particularly unpleasant. A specific example of the prompt information (message) of the alert level 1 is "Here, please avoid actions A, B, and C."

[0235] In the alert level 2 "alert", an alert for a specific action is performed. That is, the information prompting system 200 alerts the actor 203. A specific example of the prompt information (message) of the alert level 2 is "Action A becomes a disturbance, please avoid."

[0236] ​At reminder level 3, "indirect advice," a stronger reminder than at reminder level 2 is provided. For example, information presentation system 200 provides a reminder to actor 203 and indicates that there are people who are troubled by actor 203's actions, thereby indirectly conveying to actor 203 that they are taking inappropriate actions. Specific examples of reminder level 3 messages include "Someone is troubled by action A" and "Please refrain from taking action A, as it will cause trouble to other users."

[0237] At reminder level 4, "Direct Advice," a stronger reminder is provided. Specifically, the information presentation system 200 issues a strong warning, instructing the actor 203 to stop the action. Specific examples of reminder information (messages) at reminder level 4 are "Action A is prohibited here." and "Please stop action A."

[0238] For example, if the final determination probability is "high" and there are no surrounding persons 204, the actor 203 does not need to be concerned about the surrounding eyes even if he is reminded. Therefore, a direct reminder level of 4 with a high reminder degree is selected.

[0239] For example, if the final determination probability is "high" and there are surrounding persons 204, when the actor 203 is directly reminded, he or she may be concerned about the surrounding eyes and feel embarrassed. Therefore, reminder level 3 is selected for indirectly giving a reminder to the actor 203.

[0240] For example, when the final determination possibility is “suspicious”, in order to inform people around that the action of the actor 203 may cause trouble, the reminder level 2 is selected.

[0241] For example, when the final determination probability is "low", the reminder level 1 which is a general reminder level for enlightenment is selected.

[0242] exist Figure 17 The flowchart of step S132 is shown in FIG.

[0243] In step S1321 , the level determination unit 131 checks the final determination possibility.

[0244] If the final determination possibility is "low", the process proceeds to step S1322.

[0245] If the final determination possibility is "suspicious", the process proceeds to step S1323.

[0246] If the final determination possibility is "high", the process proceeds to step S1324.

[0247] In step S1322 , the level determination unit 131 selects reminder level 1 .

[0248] After step S1322, step S132 ends.

[0249] In step S1323 , the level determination unit 131 selects reminder level 2 .

[0250] After step S1323, step S132 ends.

[0251] In step S1324 , the level determination unit 131 checks the presence or absence of the surrounding person 204 .

[0252] If the surrounding person 204 exists, the process proceeds to step S1325.

[0253] If there is no surrounding person 204, the process proceeds to step S1326.

[0254] In step S1325 , the level determination unit 131 selects reminder level 3 .

[0255] After step S1325, step S132 ends.

[0256] In step S1326 , the level determination unit 131 selects reminder level 4 .

[0257] After step S1326, step S132 ends.

[0258] Alternatively, the level determination unit 131 may determine the warning level based on any one of the degree of bother of the corresponding action, the consistency level of the corresponding action with respect to the violating action, and the presence or absence of the surrounding person 204 , or any combination thereof.

[0259] return Figure 14 , and continue the description from step S133.

[0260] In step S133, the information selection unit 132 selects the reminder information corresponding to the reminder level from the plurality of reminder information. The selected reminder information becomes the reminder information.

[0261] The reminder information is predetermined as information to be presented to the actor 203 , and is stored in advance in the storage unit 190 in association with the reminder level.

[0262] Step S133 will be described in detail.

[0263] The information selection unit 132 selects reminder information corresponding to at least one of the reminder level, the type of corresponding action, and the type of reminder device.

[0264] Step S133 will be described in detail.

[0265] The information selection unit 132 selects the reminder information data 197 (see Figure 18 ) selects a reminder message corresponding to the reminder level, the type of reminder device, and the type of corresponding action. The selected reminder message is the reminder message.

[0266] exist Figure 18 2 shows the reminder information data 197. The reminder information data 197 is stored in the storage unit 190 in advance.

[0267] The reminder information data 197 shows reminder information for each combination of a reminder level, a type of reminder device, and a type of corresponding action.

[0268] exist Figure 18 In FIG. 2 , “information D” is reminder information for the display 221 , “information P” is reminder information for the projector 223 , and “information S” is reminder information for the speaker 222 .

[0269] Furthermore, the action type "none" indicates that the final determination probability is "low." When the final determination probability is "low," the warning level is "1."

[0270] The reminder information data 197 indicates information D, information P, and information S for the action type "none." That is, the reminder information data 197 indicates three types of reminder information for the action type "none."

[0271] Reminder information data 197 shows, for action A, information D for reminder level 2, information P for reminder level 2, and information S for reminder level 2. Furthermore, reminder information data 197 shows, for action A, information D for reminder level 3, information P for reminder level 3, and information S for reminder level 3. Furthermore, reminder information data 197 shows, for action A, information D for reminder level 4, information P for reminder level 4, and information S for reminder level 4. In other words, reminder information data 197 shows nine types of reminder information for action A.

[0272] Likewise, the reminder information data 197 shows nine types of reminder information for action B and nine types of reminder information for action C.

[0273] Therefore, the reminder information data 197 indicates a total of 30 types of reminder information.

[0274] Supplement the reminder information of each reminder level.

[0275] Level 1 reminders (information) include, for example, information about prohibited behaviors such as "looking at your phone while walking," "running," and "smoking." Specifically, Level 1 reminders are designed to be non-offensive even in public spaces.

[0276] The reminder information for reminder level 1 may have the same content regardless of the type of corresponding action.

[0277] The warning message for Warning Level 2 is selected when the final judgment probability is "suspicious." If the final judgment probability is "suspicious," the actor 203 has not actually committed any illegal activity. Therefore, if a warning message with strong content is presented to the actor 203, it may cause discomfort to the actor 203 who has not committed any illegal activity. Therefore, it is desirable that the warning message for Warning Level 2 be of such content that even if presented to many users 202, including the actor 203, it is unlikely to cause discomfort to each user 202.

[0278] For example, the Level 2 reminder information specifies the type of violation corresponding to the action and encourages users not to commit the specified violation. By specifying the type of violation, the user 203, who is believed to be committing the violation, is expected to recognize that their actions violate etiquette. Furthermore, the user 202 is expected to receive the information presented as general enlightenment information, similar to the Level 1 reminder information.

[0279] The reminder message for Reminder Level 3 is selected when the final determination probability is "high" and there are nearby persons 204. When a strong reminder message, such as a direct warning to the actor 203 to stop the action, is given, there is a high probability that nearby persons 204 will also notice the reminder message. Furthermore, once the actor 203 is identified, the actor 203 may feel embarrassed and uncomfortable. Therefore, the reminder message for Reminder Level 3 serves as an indirect reminder to the actor 203.

[0280] By indirectly notifying that "a specific illegal action has occurred" and "other users 202 are troubled by the illegal action", it is expected that the actor 203 who believes that the illegal action is being performed will voluntarily correct the action.

[0281] The warning message for warning level 4 is selected when the final determination probability is "high" and there are no nearby persons 204. If there are no nearby persons, even if a direct warning is given to the actor 203, the actor 203 is less likely to feel embarrassed. Therefore, the warning message for warning level 4 is a direct warning message that requires the offending behavior to be stopped.

[0282] The reminder information can be appropriately defined in a manner that gradually intensifies the degree of the reminder according to the reminder level. For example, the reminder information for each reminder level can be defined in a manner that gradually intensifies the tone of the message. In addition, if the degree of the reminder is desired to be intensified, the reminder information can be defined in a manner that emits a warning sound along with the reminder message. Furthermore, the reminder information can be defined in a manner that causes the lighting device 224 to flash or changes the lighting color of the lighting device 224. Reminder information for situations where there are people around 204 can be defined as indirect reminder content.

[0283] However, the information selection unit 132 may select reminder information corresponding to any one of the reminder level, the type of reminder device, and the type of corresponding action, or any combination thereof.

[0284] return Figure 14 , step S134 is explained.

[0285] In step S134 , the information presenting unit 134 presents the presentation information using the presentation device.

[0286] For example, the information presenting unit 134 sends the presentation information to the presentation device, and the presentation device receives the presentation information and outputs the received presentation information.

[0287] For example, the display 221 receives a text message or a message represented by an animation as prompt information, and displays the message on the screen.

[0288] For example, the speaker 222 receives a voice message as a prompt and outputs the voice message. The speaker 222 may also be a directional speaker, with the speaker 222 outputting the voice message toward the action location. In this case, the voice message is easily heard by the actor 203 but difficult to be heard by other users 202.

[0289] For example, the projector 223 receives a text message or a message represented by an animation as prompt information, and projects the message onto the ground.

[0290] For example, the lighting device 224 receives prompt information indicating a lighting pattern, and lights up, flashes, or changes the lighting color according to the lighting pattern.

[0291] **Effects of Implementation Method 1**

[0292] In a public space (monitored location 201) used by an unspecified number of people, the information presentation system 200 detects violators (actors 203) who violate etiquette and warns the violators, thereby preventing trouble between people when violators are warned.

[0293] The information prompting system 200 maintains multiple reminder messages with varying degrees of severity. Furthermore, the information prompting system 200 appropriately changes the reminder message presented based on the likelihood of the violation and the violator's surroundings (presence of other people, level of distress). Therefore, even in a public space used by a large number of people, the violator receiving the reminder will not feel uncomfortable or embarrassed. Furthermore, effective reminder information tailored to the situation can be provided.

[0294] Implementation method 2.

[0295] Regarding the method for detecting two or more corresponding actions, the differences from the first embodiment will be mainly described.

[0296] ***Structure description***

[0297] The structure of the information presentation system 200 is similar to that in the first embodiment (see Figures 1 to 3 )same.

[0298] ***Action description***

[0299] based on Figure 5 The behavior analysis ( S110 ) will be described.

[0300] In step S111, the action detection unit 111 detects two or more corresponding actions and specifies the type of each corresponding action.

[0301] Furthermore, the action detection unit 111 identifies the actor 203 for each corresponding action and calculates the position of the actor 203 .

[0302] In step S112 , the action determination unit 112 calculates the degree of consistency between the corresponding action and the violation action for each corresponding action.

[0303] Then, the action determination unit 112 determines the corresponding action with the highest degree of consistency as the prioritized corresponding action.

[0304] For example, when the degree of consistency of the corresponding action of actor A is “70%” and the degree of consistency of the corresponding action of actor B is “50%”, the action determination unit 112 determines the corresponding action of actor A as the prioritized corresponding action.

[0305] When there are two or more corresponding actions with the highest degree of consistency, the action determination unit 112 determines any one of the two or more corresponding actions with the highest degree of consistency as a priority corresponding action based on the respective types of the two or more corresponding actions with the highest degree of consistency.

[0306] Specifically, the priority of each violation action is predefined for a plurality of violation actions. Then, the action determination unit 112 determines the action with the highest priority as the priority action from among the two or more corresponding actions with the highest degree of consistency.

[0307] For example, the reminder information data 197 ( Figure 18 ) is the priority order of the violating actions. Here, it is assumed that the consistency of the corresponding action of actor A is the same as the consistency of the corresponding action of actor B. If the type of the corresponding action of actor A is "Action A" and the type of the corresponding action of actor B is "Action B", the action determination unit 112 determines that the corresponding action of actor A is the priority corresponding action.

[0308] The subsequent processing (step S113, step S120, and step S130) is executed for the prioritized corresponding action.

[0309] In step S121 (refer to Figure 8 ), the situation detection unit 121 may exclude the actors 203 of other corresponding actions from the surrounding persons 204 when determining the presence and number of the surrounding persons 204 of the priority corresponding action.

[0310] **Effects of Implementation Method 2**

[0311] According to the second embodiment, when two or more corresponding actions are detected, the corresponding action to be prioritized can be identified and a reminder can be issued.

[0312] Implementation method 3.

[0313] Regarding the method of detecting the corresponding action using the sound data, the differences from the first embodiment will be mainly described.

[0314] Specific examples of violations involving sound include making sounds in places where silence is required (such as libraries and art galleries) or in public enclosed spaces (such as inside trains, buses, and elevators).

[0315] ***Structure description***

[0316] The structure of the information presentation system 200 is similar to that in the first embodiment (see Figures 1 to 3 ).

[0317] ***Action description***

[0318] based on Figure 19 The information prompt method is explained.

[0319] In step S310 , the behavior analysis unit 110 performs behavior analysis.

[0320] Step S310 corresponds to step S110 in Embodiment 1.

[0321] Step S310 will be described in detail later.

[0322] Step S120 and step S130 are the same as steps in Embodiment 1.

[0323] Based on Figure 20 The action analysis (S310) will be described.

[0324] Steps S311 to S313 correspond to steps S111 to S113 in Embodiment 1.

[0325] In step S311, the action detection section 111 detects the corresponding action by analyzing the sound data collected by monitoring the sound in the site 201.

[0326] The type of the violation action detected as the corresponding action is "vocalization that becomes noise".

[0327] Specifically, the action detection section 111 detects the voice by analyzing the sound data, measures the volume of the detected voice, and detects the corresponding action based on the measured volume.

[0328] In the feature data set 191 (refer to Figure 6 ), the feature data of the type of the violation action such as "vocalization that becomes noise" is included. The volume threshold value is included in the feature information indicated by the feature data.

[0329] In a case where the measured volume is equal to or higher than the volume threshold value, the action detection section 111 detects the vocalization that emits the detected voice as the corresponding action.

[0330] Further, the action detection section 111 acquires the position information of the microphone 212 that collected the sound data from the storage section 190. The position information of the microphone 212 is stored in the storage section 190 in advance.

[0331] The position of the microphone 212, that is, the sound pickup position, becomes the action position.

[0332] In step S312, the action determination section 112 calculates the difference between the volume of the corresponding action and the volume threshold value. The calculated difference is referred to as the exceeding volume. The exceeding volume corresponds to the degree of agreement.

[0333] In step S313, the action determination section 112 determines the violation action possibility of the corresponding action with respect to the violation action such as "vocalization that becomes noise" based on the exceeding volume.

[0334] The likelihood of a violation of an action is equivalent to the volume level of the action. The volume level is equivalent to the consistency level.

[0335] based on Figure 8 The situation analysis ( S120 ) will be described.

[0336] In step S121 , the situation detection unit 121 analyzes the image data and determines the presence or absence of the surrounding person 204 .

[0337] One of the users 202 located around the sound pickup position is an actor 203 .

[0338] Among the users 202 located around the sound pickup position, users 202 other than the actor 203 are surrounding persons 204 .

[0339] When there are two or more users 202 around the sound pickup position, there are surrounding persons 204 .

[0340] When there is only one user 202 around the sound pickup position, there is no surrounding person 204 .

[0341] The subsequent processing (step S122, step S123, and step S130) is the same as that in the first embodiment.

[0342] However, the selection rule may be defined to preferentially select the display 221 or the projector 223 as the prompting device. Furthermore, the lighting device 224 near the action location may be selected as the prompting device along with the display 221 or the projector 223, and the lighting device 224 may be lit or flashed to make it easier for the actor 203 to notice the prompting information.

[0343] **Effects of Implementation Method 3**

[0344] According to the third embodiment, it is possible to detect corresponding actions using audio data and issue a reminder.

[0345] Implementation method 4.

[0346] Regarding the method of detecting a violation such as “crowded people”, the differences from the first embodiment will be mainly described.

[0347] ***Structure description***

[0348] The structure of the information presentation system 200 is similar to that in the first embodiment (see Figures 1 to 3 )same.

[0349] ***Action description***

[0350] based on Figure 21 The information prompt method is explained.

[0351] In step S410 , the behavior analysis unit 110 performs behavior analysis.

[0352] Step S410 corresponds to step S110 in the first embodiment.

[0353] based on Figure 22 The behavior analysis ( S410 ) will be described.

[0354] Steps S411 to S413 correspond to steps S111 to S113 in the first embodiment.

[0355] In step S411 , the action detection unit 111 analyzes the image data and detects corresponding actions.

[0356] For example, the action detection unit 111 detects an action of the type “crowded with people.” In this case, it is not necessary to identify the actor 203 or calculate the position of the actor 203 .

[0357] The corresponding behavior of the type "people crowding" is detected as follows.

[0358] First, the behavior detection unit 111 detects each user 202 appearing in the video data.

[0359] Next, the behavior detection unit 111 counts the number of users 202 .

[0360] Then, the behavior detection unit 111 compares the number of users 202 with the number threshold.

[0361] When the number of people exceeds the threshold, a corresponding action of the type "crowded with people" occurs.

[0362] The number threshold is included in the characteristic data for the violation action type "crowd of people" (see Figure 6 ) in the characteristic information. “Number of people” may be replaced with “density.” Density is calculated using the number of users 202 and the area of ​​the monitored location 201.

[0363] When the type of the corresponding behavior is other than "people crowding", steps S412 and S413 are the same as steps S112 and S113 in the first embodiment.

[0364] The following describes a case where the type of corresponding action is "human density."

[0365] In step S412, the action determination unit 112 calculates the difference between the number of users 202 and the number threshold. The calculated difference is referred to as the excess number of users. The excess number of users corresponds to the degree of consistency.

[0366] In step S413 , the action determination unit 112 determines the possibility of a violation action of the corresponding action with respect to the violation action of “crowding of people” based on the number of people exceeded.

[0367] The violation probability of the corresponding action is equivalent to the density level of the corresponding action. The density level is equivalent to the consistency level.

[0368] return Figure 21 , and the description continues from step S420.

[0369] In step S420 , the action determination unit 112 confirms the type of the corresponding action.

[0370] When the type of the corresponding action is "people densely packed", the actor 203 and the action position are not identified, and detection of the surrounding conditions and determination of the distress level are not necessary, so the process proceeds to step S130.

[0371] If the type of the corresponding action is other than "crowded with people", the process proceeds to step S120.

[0372] Step S120 and step S130 are the same as those in the first embodiment.

[0373] However, if the type of the corresponding action is "people crowded", in step S132 (refer to Figure 14 ), the level determination unit 131 determines the reminder level based not on the degree of trouble but on the possibility of the violation action (the density level).

[0374] Specific examples of presentation information (messages) at each warning level for “crowded areas” are shown below.

[0375] A specific example of the reminder level 1 message is “Pay attention to social distance (social distance)”.

[0376] A specific example of a message at alert level 2 is "Keep a distance to prevent infection."

[0377] A specific example of a message at alert level 3 is “Please maintain distance to prevent infection.”

[0378] A specific example of a message of alert level 4 is “A close contact state has occurred. Please keep some distance.”

[0379] **Effects of Implementation 4**

[0380] According to the fourth embodiment, it is possible to detect violations such as "crowding" and issue a warning. Therefore, it is possible to prevent the density of people from increasing. In other words, the fourth embodiment is effective for responding to infectious diseases such as influenza and other viruses.

[0381] Implementation method 5.

[0382] Regarding the method of confirming whether the corresponding action has been corrected after the reminder information is presented, the differences from the first embodiment will be mainly described.

[0383] ***Structure description***

[0384] The structure of the information presentation system 200 is similar to that in the first embodiment (see Figures 1 to 3 )same.

[0385] ***Action description***

[0386] based on Figure 23 The information prompt method is explained.

[0387] In step S501 , the action detection unit 111 sets the number of presentations n to zero.

[0388] The number of reminders n is a variable used to manage the number of reminders. The initial value is zero.

[0389] In step S502 , the action detection unit 111 increments the number of presentations n by 1.

[0390] Step S110 and step S120 are the same as those in the first embodiment.

[0391] In step S530 , the presentation control unit 130 performs presentation control.

[0392] In the presentation control, the level determination unit 131 determines the reminder level based on the number of presentations and the degree of distress.

[0393] Furthermore, the information presenting unit 134 determines whether the number of presentations satisfies a presentation condition, and presents the presentation information if the number of presentations satisfies the presentation condition.

[0394] Step S530 will be described in detail later.

[0395] In step S503 , the action detection unit 111 compares the number of times n presented with the upper limit number Y. The upper limit number Y is a number that is predetermined as the upper limit of the number of times presented. For example, the upper limit number Y is 4.

[0396] When the number of presentations n is zero or equal to or greater than the upper limit number Y, the process ends.

[0397] When the number of presentations n is equal to or greater than 1 and less than the upper limit number Y, the process proceeds to step S502.

[0398] based on Figure 24 The prompt control (S530) will be described.

[0399] In step S531 , the device selection unit 133 selects a presentation device.

[0400] Step S531 is the same as step S131 in the first embodiment.

[0401] In step S532 , the level determination unit 131 determines the warning level based on the degree of bother of the corresponding action, the consistency level of the corresponding action with respect to the violating action, the presence or absence of the surrounding person 204 , and the number of prompts n.

[0402] Step S532 will be described in detail. The level determination unit 131 performs the processes (1) and (2). The likelihood of a violation based on the degree of consistency corresponds to the degree of consistency.

[0403] (1) As described in the first embodiment, the level determination unit 131 determines the final determination possibility based on the degree of bothersomeness of the corresponding action and the possibility of the violation action based on the degree of consistency.

[0404] (2) The level determination unit 131 determines the warning level based on the final determination probability, the presence or absence of the surrounding person 204 , and the number of prompting times n.

[0405] based on Figure 25 Step S532 (2) will be described in detail.

[0406] In step S5321 , the level determination unit 131 checks the final determination possibility.

[0407] If the final determination possibility is "low", the process proceeds to step S5322.

[0408] When the final determination possibility is "suspicious", the process proceeds to step S5324.

[0409] If the final determination possibility is "high", the process proceeds to step S5325.

[0410] In step S5322 , the level determination unit 131 selects reminder level 1 .

[0411] In step S5323 , the level determination unit 131 sets the number of presentations n to zero.

[0412] After step S5323, the processing ends.

[0413] In step S5324 , the level determination unit 131 selects a reminder level based on the number of presentations n.

[0414] Specifically, the level determination unit 131 selects a reminder level corresponding to the number of presentations n from the reminder level data 198A.

[0415] exist Figure 26 2 shows the warning level data 198A. The warning level data 198A is stored in the storage unit 190 in advance.

[0416] The reminder level data 198A associates the number of reminders n with the reminder level according to different reminder times n.

[0417] For example, when the number of presentations n is 1, that is, when the reminder information is presented for the first time, the level determination unit 131 selects the reminder level 1.

[0418] Furthermore, when the number of presentations n is 2 or greater, that is, when the reminder information is presented for the second time or later, the level determination unit 131 selects the reminder level 2 .

[0419] In step S5325 , the level determination unit 131 checks the presence or absence of the surrounding person 204 .

[0420] If the surrounding person 204 exists, the process proceeds to step S5326.

[0421] If there is no surrounding person 204, the process proceeds to step 5327.

[0422] In step S5326 , the level determination unit 131 selects a reminder level based on the number of presentations n.

[0423] Specifically, the level determination unit 131 selects a reminder level corresponding to the number of presentations n from the reminder level data 198B.

[0424] exist Figure 27 The warning level data 198B is shown in . The warning level data 198B is stored in the storage unit 19 in advance.

[0425] The reminder level data 198B associates the number of reminders n with the reminder level according to different reminder times n.

[0426] For example, when the number of presentations n is 1, that is, when the reminder information is presented for the first time, the level determination unit 131 selects reminder level 1.

[0427] Furthermore, when the number of presentations n is 2, that is, when the reminder information is presented for the second time, the level determination unit 131 selects reminder level 2.

[0428] Furthermore, when the number of presentations n is 3 or greater, that is, when the reminder information is presented for the third time or later, the level determination unit 131 selects reminder level 3 .

[0429] In step S5327 , the level determination unit 131 selects a reminder level based on the number of presentations n.

[0430] Specifically, the level determination unit 131 selects a reminder level corresponding to the number of reminders n from the reminder level data 198C.

[0431] exist Figure 28 2 shows the warning level data 198C. The warning level data 198C is stored in the storage unit 190 in advance.

[0432] The reminder level data 198C associates the number of reminders n with the reminder level according to different reminder times n.

[0433] For example, when the number of presentations n is 1, that is, when the reminder information is presented for the first time, the level determination unit 131 selects reminder level 1.

[0434] Furthermore, when the number of presentations n is 2, that is, when the reminder information is presented for the second time, the level determination unit 131 selects reminder level 2.

[0435] Furthermore, when the number of presentations n is 3, that is, when the reminder information is presented for the third time, the level determination unit 131 selects reminder level 3.

[0436] Furthermore, when the number of presentations n is 4 or greater, that is, when the reminder information is presented for the fourth time or later, the level determination unit 131 selects reminder level 4 .

[0437] return Figure 24 , and the description continues from step S533.

[0438] In step S533 , the information selection unit 132 selects presentation information corresponding to the reminder level.

[0439] Step S533 is the same as step S133 in the first embodiment.

[0440] In step S534 , the information presenting unit 134 confirms the number of presentations n.

[0441] When the number of presentations n is zero or equal to or greater than the upper limit number Y, the process proceeds to step S535.

[0442] When the number of presentations n is equal to or greater than 1 and less than the upper limit number Y, the presentation information is not presented and the process ends.

[0443] In step S535 , the information presentation unit 134 presents the presentation information using the presentation device.

[0444] Step S535 is the same as step S134 in the first embodiment.

[0445] **Effects of Implementation 5**

[0446] According to the fifth embodiment, it is possible to confirm whether the corresponding action has been corrected after the reminder information is presented.

[0447] In the case that the corresponding action has not been corrected, the information prompting system 200 prompts a reminder message with a stronger reminder level than the last reminder message.

[0448] That is, if the probability of a violation does not become "low" after the initial reminder message is presented, the information presentation system 200 repeatedly presents the reminder message. Therefore, even if the first reminder does not lead to corrective action, the possibility of corrective action can be increased through multiple reminders.

[0449] Furthermore, because the reminder information presented gradually becomes stronger, actor 203 is more likely to accept the reminder information, and the likelihood of correcting their behavior can be expected to increase. The reason for this is as follows. If actor 203 is unaware that they are in the process of violating the rules, a strong reminder from the beginning will make them unclear about the reason for the reminder and likely cause them discomfort. Therefore, by gradually increasing the level of reminder information from a low level, actor 203 gradually becomes aware of the ongoing violation. This makes actor 203 more likely to accept the reminder information, and the likelihood of them voluntarily correcting their behavior increases.

[0450] **Supplement to the implementation method**

[0451] based on Figure 29 The hardware structure of the information presentation device 100 will be described.

[0452] The information presentation device 100 includes a processing circuit 109 .

[0453] The processing circuit 109 is hardware that realizes the behavior analysis unit 110 , the situation analysis unit 120 , and the presentation control unit 130 .

[0454] The processing circuit 109 may be dedicated hardware or the processor 101 that executes a program stored in a memory.

[0455] In the case where the processing circuit 109 is dedicated hardware, the processing circuit 109 is, for example, a single circuit, a complex circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0456] ASIC is the abbreviation of Application Specific Integrated Circuit.

[0457] FPGA is the abbreviation of Field Programmable Gate Array.

[0458] The information presentation device 100 may include a plurality of processing circuits instead of the processing circuit 109 .

[0459] In the processing circuit 109 , part of the functions may be implemented by dedicated hardware, and the remaining functions may be implemented by software or firmware.

[0460] In this way, the functions of the information presentation device 100 can be implemented by hardware, software, firmware, or a combination thereof.

[0461] Each embodiment is an illustration of a preferred mode and is not intended to limit the technical scope of the present disclosure. Each embodiment can be implemented in part or in combination with other modes. The steps described using flowcharts and the like can also be appropriately changed.

[0462] The information display device 100 may also be implemented by multiple devices.

[0463] The “unit” as an element of the information presentation device 100 may be replaced with a “process,” “processing,” “circuit,” or “line.”

[0464] Description of Reference Numerals

[0465] 100 Information presentation device, 101 Processor, 102 Main storage device, 103 Auxiliary storage device, 104 Input interface, 105 Output interface, 106 Communication device, 109 Processing circuit, 110 Action analysis unit, 111 Action detection unit, 112 Action determination unit, 120 Situation analysis unit, 121 Situation detection unit, 122 Notification acceptance unit, 123 Trouble level determination unit, 130 Presentation control unit, 131 Level determination unit, 132 Information selection unit, 133 Device selection unit, 134 Information presentation unit, 190 Storage unit, 191 Feature data group, 192 Violation action possibility data, 19 3 Surrounding condition data group, 194A Expression distress data, 194B Action distress data, 194C Notification distress data, 194D Comprehensive distress data, 195 Final judgment probability data, 196 Reminder level data, 197 Reminder information data, 198 Reminder level data, 200 Information notification system, 201 Monitoring location, 202 User, 203 Actor, 204 Surrounding persons, 210 Monitoring equipment, 211 Camera, 212 Microphone, 220 Output device, 221 Display, 222 Speaker, 223 Projector, 224 Lighting equipment, 231 Portable terminal.

Claims

1. An information prompting system comprising: an action detection unit that analyzes surveillance data obtained by monitoring a surveillance location and detects corresponding actions carried out at the surveillance location and having characteristics common to violations; a situation detection unit that analyzes image data obtained by photographing the monitored location and detects a surrounding situation of an action location corresponding to a location where the corresponding action is performed, wherein information indicating the surrounding situation includes at least one of the presence or absence of a person around the action location, an expression of the person around the action location, and an action of the person around the action location; a distress level determination unit configured to determine a distress level of the corresponding action based on the surrounding conditions; a level determination unit configured to determine a warning level based on the degree of distress; as well as An information presenting unit presents the presenting information corresponding to the warning level using a presenting device installed at the monitoring location.

2. The information prompting system according to claim 1, wherein: The action detection unit detects the corresponding action by analyzing the image data as the surveillance data, identifies the actor who performed the corresponding action, and calculates the position of the actor. The situation detection unit detects the surrounding situation using the position of the actor as the action position.

3. The information prompting system according to claim 1, wherein: The distress level determination unit determines the distress level based on at least one of an expression of the surrounding person and an action of the surrounding person.

4. The information prompting system according to claim 2, wherein: The distress level determination unit determines the distress level based on at least one of an expression of the surrounding person and an action of the surrounding person.

5. The information prompting system according to any one of claims 1 to 4, wherein: The information presentation system includes a notification receiving unit that receives trouble notifications sent from respective mobile terminals of persons around the activity location. The trouble level determination unit determines the trouble level based on the surrounding conditions and the number of trouble notifications.

6. The information prompting system according to any one of claims 1 to 4, wherein: The information prompting system includes an action determination unit that calculates the consistency between the corresponding action and the violating action based on the characteristic information of the violating action and the characteristic information of the corresponding action obtained from the monitoring data, and determines the consistency level based on the calculated consistency. The level determination unit determines the reminder level based on the degree of distress and the consistency level.

7. The information prompting system according to claim 6, wherein: The level determination unit determines the warning level based on the degree of trouble, the consistency level, and the presence or absence of the surrounding person.

8. The information prompting system according to any one of claims 1 to 4, wherein: The information prompting system includes an information selection unit that selects the prompting information corresponding to the prompting level from a plurality of prompting information. The action detection unit detects the corresponding actions for a plurality of violation actions. The information selection unit selects, as the presentation information, reminder information corresponding to the reminder level and the type of the corresponding action, which is the type of the violation action corresponding to the corresponding action.

9. The information prompting system according to claim 8, wherein: A variety of output devices are provided in the monitoring location. The information presentation system includes a device selection unit that selects an output device to be used as the presentation device from among the plurality of output devices based on at least one of the action position and the surrounding conditions. The information selection unit selects, as the presentation information, the presentation information corresponding to the presentation level, the type of the corresponding action, and the type of the presentation device, which is the type of the output device also used as the presentation device.

10. The information prompting system according to any one of claims 1 to 4, wherein: A variety of output devices are provided in the monitoring location. The information presentation system includes a device selection unit that selects an output device to be used as the presentation device from among the plurality of output devices based on at least one of the action position and the surrounding conditions.

11. The information prompting system according to claim 1, wherein: The information prompting system includes an action determination unit, The action detection unit detects the corresponding actions for a plurality of violation actions. When the action determination unit detects two or more corresponding actions corresponding to two or more violation actions, it determines the consistency between the corresponding action and the violation action based on the characteristic information of the violation action corresponding to the corresponding action and the characteristic information of the corresponding action obtained from the monitoring data for each corresponding action, and determines the corresponding action with the highest consistency as the priority corresponding action.

12. The information prompting system according to claim 11, wherein: When there are two or more corresponding actions with the highest degree of consistency, the action determination unit determines any one of the two or more corresponding actions with the highest degree of consistency as the priority corresponding action based on the respective types of the two or more corresponding actions with the highest degree of consistency.

13. The information prompting system according to claim 1, wherein: The action detection unit analyzes the sound data obtained by collecting sounds in the monitoring location as the monitoring data, and detects the corresponding action by making a sound that becomes noise as the violation action. The situation detection unit detects the surrounding situation using a sound pickup position in the monitoring location as the action position.

14. The information prompting system according to claim 13, wherein: The information prompting system includes an action determination unit, The action detection unit analyzes the audio data and measures the volume of the voice, and detects the corresponding action based on the measured volume. The action determination unit determines the volume level of the corresponding action based on the volume of the corresponding action, The level determination unit determines the warning level based on the degree of distress and the volume level.

15. The information prompting system according to claim 14, wherein: The situation detection unit detects whether there are any surrounding persons other than the person who has performed the corresponding action and is located around the sound collection position as one of the information indicating the surrounding situation. The level determination unit determines the warning level based on the degree of annoyance, the volume level, and the presence or absence of the surrounding person.

16. The information prompting system according to claim 1, wherein: The information prompting system includes an action determination unit, The action detection unit analyzes the image data as the surveillance data, detects each person shown in the image, and detects the corresponding action for the violation action of the type of crowding of people based on the number of detected people. When the action determination unit detects the corresponding action in response to the violation action of the type of people being crowded, the action determination unit determines the level of the density of the corresponding action based on the number of the detected people. The level determination unit determines the warning level based on the degree of annoyance when the corresponding action is detected for a violation other than the human crowding, and determines the warning level based on the crowding level when the corresponding action is detected for a violation such as the human crowding.

17. The information prompting system according to any one of claims 1 to 4, wherein: The level determination unit determines the reminder level based on the degree of distress and the number of times the reminder information is presented.

18. The information prompting system according to claim 17, wherein: The information presenting unit presents the presentation information when the number of presentations satisfies a presentation condition, and does not present the presentation information when the number of presentations does not satisfy the presentation condition.

19. An information prompting method, wherein: The action detection unit analyzes monitoring data obtained by monitoring a monitoring location and detects corresponding actions that are carried out at the monitoring location and have characteristics common to the illegal actions. The situation detection unit analyzes the image data obtained by photographing the surveillance location and detects the surrounding situation of the action location corresponding to the location where the corresponding action was performed, wherein the information indicating the surrounding situation includes at least one of the presence or absence of a person around the action location, the expression of the person around the action location, and the action of the person around the action location. The trouble level determination unit determines the trouble level of the corresponding action based on the surrounding conditions. The level determination unit determines a warning level based on the degree of distress. The information presenting unit presents presentation information corresponding to the warning level using a presentation device installed at the monitoring location.

20. A computer-readable recording medium having an information prompting program recorded thereon, the information prompting program being configured to cause a computer to execute the following processing: Action detection processing, analyzing surveillance data obtained from monitoring a surveillance location, detecting corresponding actions carried out at the surveillance location and having characteristics common to illegal actions, Situation detection processing includes analyzing image data obtained by photographing the surveillance location to detect a surrounding situation of an action location corresponding to the location where the corresponding action was performed, wherein information indicating the surrounding situation includes at least one of the presence or absence of surrounding persons around the action location, the expressions of the surrounding persons, and the actions of the surrounding persons. a distress level determination process for determining the distress level of the corresponding action based on the surrounding conditions; Level determination processing, determining the reminder level based on the degree of distress, The information prompting process uses a prompting device installed at the monitoring location to prompt prompting information corresponding to the warning level.

21. A disturbance level determination system comprising: An action detection unit, which uses surveillance data obtained from monitoring a surveillance location to detect actions that may cause distress to others; a trouble level determination unit that determines a trouble level of the corresponding action based on a surrounding condition of an action location where the corresponding action is performed, detected based on image data obtained by photographing the monitored location; and a situation detection unit that detects at least one of an expression of a surrounding person located around the action position and an action of the surrounding person as one of the information indicating the surrounding situation; The distress level determination unit selects a distress level based on the expression of the surrounding person from the expression distress level data, selects a distress level based on the behavior of the surrounding person from the behavior distress level data, and selects a comprehensive distress level corresponding to a combination of the selected distress level based on the expression and the distress level based on the behavior from the comprehensive distress level data as the distress level of the corresponding behavior.

Images