Information aggregation system and information aggregation method

Abstract

To provide a mechanism configured to take measures against an event even if it is not reported for accident.SOLUTION: An information aggregation system includes: a first storage unit which stores specific action information including types of specific actions, which are actions to be detected, out of risky actions against which measures are to be formulated in advance, and information representing the change in attitude due to the specific actions; a map information storage unit which stores map information of a region in which the specific actions are to be detected; an image acquisition unit which acquires a captured image of the detection region; an action detection unit which detects a specific action from the image acquired by the image acquisition unit, based on the specific action information; a location estimation unit which estimates a location where the detected specific action occurred in the region indicated in the map information, on the basis of the image acquired by the image acquisition unit; a second storage unit which stores information on the type of the specific action detected by the action detection unit in association with the location estimated by the location estimation unit; and an output unit which outputs a result based on the information stored in the second storage unit.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information aggregation system and an information aggregation method.

Background Art

[0002] Conventionally, in companies and the like, when an accident (injury) such as an industrial accident occurs, a report is made about the details of the accident. When a report is made, measures are taken to prevent the occurrence of the same accident. Patent Document 1 discloses a technique for preventing the occurrence of accidents by accumulating and sharing records of accidents that have occurred so far.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, regarding actions that do not result in accidents but have the potential to lead to accidents, such as falls and collisions, reports are not made and thus do not remain in the records. For example, in most cases, actions such as tripping without actually falling are not reported. Events for which no reports have been made have been excluded from the targets for taking measures.

[0005] Therefore, the present invention has been made in view of the above points, and provides a mechanism capable of taking measures even for events for which no reports regarding accidents have been made.

Means for Solving the Problems

[0006] One aspect of the present invention is a first storage unit that stores specific action information including the type of a specific action that is an action to be detected among attention actions for which countermeasures are to be taken in advance, and information representing a change in posture due to the specific action; a map information storage unit that stores map information of an area for detecting the specific action; an image acquisition unit that acquires an image of the area to be detected; an action detection unit that detects the specific action from the image acquired by the image acquisition unit based on the specific action information; a position estimation unit that estimates a position where the detected specific action has occurred in the area to be detected indicated by the map information based on the image acquired by the image acquisition unit; a second storage unit that stores in association information regarding the type of the specific action detected by the action detection unit and the position estimated by the position estimation unit; and an output unit that outputs a result based on the information stored in the second storage unit.

[0007] Further, one aspect of the present invention is, for a specific action that is an action to be detected among attention actions for which countermeasures are to be taken in advance, an image acquisition step of acquiring an image of an area for detecting the specific action; an action detection step of detecting the specific action from the image acquired in the image acquisition step based on the type of the specific action and specific action information including information indicating characteristics of an appearance represented by the specific action and a change in the position where a person is located; a position estimation step of estimating a position where the detected specific action has occurred in the area to be detected indicated by the map information of the area for detecting the specific action based on the image acquired in the image acquisition step; and an output step of outputting a result based on information associating information regarding the type of the specific action detected in the action detection step and the position estimated in the position estimation step. An information aggregation method having the above.

Advantages of the Invention

[0008] According to the present invention, countermeasures can be taken even for an event for which no accident report is made.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Mode for Carrying Out the Invention

[0010] Regarding the information aggregation system according to this embodiment, preferred embodiments will be described in detail below with reference to the accompanying drawings. Note that this embodiment is not limited to these embodiments, and also includes those with various modifications or improvements. That is, the components described below include those that can be easily assumed by those skilled in the art and substantially identical ones, and the components described below can be combined as appropriate. Also, in this embodiment, various omissions, substitutions, or changes of the components can be made without departing from the gist of the present invention.

[0011] [Embodiment 1] FIG. 1 is a schematic diagram for explaining an example in the case of detecting a specific action. With reference to this figure, the mode of the information aggregation system 1 according to the embodiment will be described. In FIG. 1, an imaging device 3 is shown imaging the interior from the four corners of the room. In FIG. 1, in the room, equipment 2-1, equipment 2-2, and equipment 2-3 are installed, and a worker OP who has fallen near equipment 2-1 is shown. The worker OP is a person who is performing work. Note that equipment 2-1 to equipment 2-3 are examples of a plurality of pieces of equipment 2. When not distinguishing each of equipment 2-1 to equipment 2-3, it may be simply described as equipment 2 below.

[0012] If the fallen worker OP is not injured and does not damage the equipment 2, they often continue working without reporting to anyone. However, if the report by the fallen worker OP is not made, it is impossible to take measures against the case where there are factors inducing the fall.

[0013] The information aggregation system 1 uses the imaging device 3 to monitor whether a specific action, which is a registered action (action to be detected) among the attention actions, is being performed or not in a detection area for detecting the specific action. The attention action is an action for which countermeasures are to be taken in advance. The attention action may include, for example, actions such as falling, tripping, colliding, stumbling, or slipping, which may lead to accidents or injuries depending on the location and situation even if no actual accidents or injuries have occurred. Further, the attention action may include, for example, actions that may cause accidents due to carelessness or non-compliance with rules, such as stepping outside the pedestrian path, not wearing protective gear, walking while using a smartphone, or performing a task that is required to be performed by two or more people alone (hereinafter, may be described as prohibited actions). When a specific action occurs in the detection area, the information aggregation system 1 detects the specific action and collects information regarding the detected specific action.

[0014] FIG. 2 is a block diagram for explaining the configuration of the information aggregation system according to Embodiment 1. With reference to this figure, the information aggregation system 1 will be specifically described. The information aggregation system 1 includes an imaging device 3, a recorder 4, a temperature and humidity sensor 5, an illuminance sensor 6, a sensor management device 7, an information aggregation device 10, and a plurality of terminal devices 30. Note that the terminal device 30-1 and the terminal device 30-N are examples of the plurality of terminal devices 30, and when not distinguishing them from each other, they may be simply referred to as the terminal device 30 hereinafter. The recorder 4, the sensor management device 7, the information aggregation device 10, and the terminal device 30 are connected to each other via a network NW. The network NW is a predetermined communication network configured using a line. As an example of the network NW, the Internet can be mentioned. The predetermined communication network NW is an open network, and devices connected to the communication network NW can communicate with each other using a predetermined protocol. In the Internet layer of the communication network NW, communication using the Internet Protocol (IP) or the like is performed. The imaging device 3 and the recorder 4, the temperature and humidity sensor 5 and the sensor management device 7, and the illuminance sensor 6 and the sensor management device 7 may be connected to each other via the network NW, or may be connected so as to be communicable with each other by a wired connection.

[0015] The imaging device 3 includes an imaging unit 31 and a transmission unit 32. The imaging device 3 may be, for example, a CCD camera using a CCD (Charge Coupled Devices) image sensor, or a CMOS camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Further, the imaging device 3 may be, for example, a depth sensing camera capable of measuring the distance from the imaging device 3 to the object.

[0016] The imaging unit 31 images a predetermined angular field region in the front. The imaging unit 31 transmits information about the captured image (hereinafter, may be referred to as image information IMI) to the transmission unit 32. Note that the image may be a still image or a moving image.

[0017] The transmission unit 32 acquires the image information IMI from the imaging unit 31. The transmission unit 32 transmits the image information IMI regarding the image captured by the imaging unit 31 to the recorder 4.

[0018] The recorder 4 acquires the image information IMI from the transmission unit 32. The recorder 4 records the image captured by the imaging unit 31. That is, the recorder 4 stores the image information IMI. The recorder 4 transmits the image information IMI to the image acquisition unit 11 provided in the information aggregation device 10. Note that the recorder 4 may transmit the recorded image to the image acquisition unit 11 at a predetermined time interval, or each time it acquires from the transmission unit 32, or transmit according to an instruction from the image acquisition unit 11.

[0019] The temperature and humidity sensor 5 is installed at a predetermined position within the detection area and measures temperature and humidity. The predetermined position where the temperature and humidity sensor 5 is installed is, for example, any position within the detection area where direct sunlight does not hit and the wind flow is gentle. Specifically, the temperature and humidity sensor 5 may be installed at a position where the air-conditioning wind does not directly hit, a position far from doors and windows with little influence of outside air, a position away from the heat-emitting equipment 2, etc. The temperature and humidity sensor 5 transmits temperature information TI indicating the temperature value at the predetermined position and humidity information HI indicating the humidity value at the predetermined position to the sensor management device 7. Note that the temperature and humidity sensor 5 may be composed of a plurality of sensors.

[0020] The illuminance sensor 6 is installed at a predetermined position within the detection area and measures illuminance. The predetermined position where the illuminance sensor 6 is installed is, for example, any position within the detection area where the operator OP mainly performs work. Specifically, when working on a desk or workbench, the illuminance sensor 6 may be installed on the upper surface of the desk or workbench or at a position within 5 [cm] from the upper surface. The illuminance sensor 6 transmits illuminance information II indicating the illuminance value at the predetermined position to the sensor management device 7. Although the predetermined positions where the temperature and humidity sensor 5 and the illuminance sensor 6 are installed have been described as above, this embodiment is not limited to this example.

[0021] The sensor management device 7 acquires temperature information TI and humidity information HI from the temperature and humidity sensor 5. Further, the sensor management device 7 acquires illuminance information II from the illuminance sensor 6. The sensor management device 7 transmits the acquired temperature information TI, humidity information HI, and illuminance information II to the environmental information acquisition unit 16 included in the information aggregation device 10.

[0022] The terminal device 30 is operated by an administrator. The administrator is, for example, a person who inputs or views information in the information aggregation system 1. The terminal device 30 performs input of severity, selection of evaluation items (condition items) and display formats, and display of information generated by the information aggregation device 10 to the information aggregation device 10 according to the operation of the administrator.

[0023] The information aggregation device 10 includes an image acquisition unit 11, an action detection unit 12, a position estimation unit 13, a time information acquisition unit 14, a personal identification unit 15, an environmental information acquisition unit 16, a severity acquisition unit 17, an information processing unit 18, an output unit 19, a first storage unit 21, a map information storage unit 22, and a second storage unit 23. Each functional unit of the information aggregation device 10 may be realized using an electronic circuit as needed. Further, each functional unit may not be included in a single device, and may be in a state where the information aggregation device 10 is configured from a plurality of devices.

[0024] The image acquisition unit 11 acquires image information IMI from the recorder 4 via the network NW. The image acquisition unit 11 transmits the acquired image information IMI to the action detection unit 12.

[0025] The action detection unit 12 acquires image information IMI from the image acquisition unit 11. The action detection unit 12 detects (monitors) whether the image information IMI includes an image corresponding to a specific action. Based on the information (hereinafter sometimes referred to as specific action information) indicating the appearance features represented by the specific action and the change in the position of the person in the specific action database 211 and the acquired image information IMI, the action detection unit 12 detects that a specific action has occurred in the detection area (the area that can be imaged by the imaging device 3). The appearance features represented by the specific action may include, for example, changes in posture caused by the occurrence of specific actions such as falling, dropping, colliding, slipping, or walking while using a smartphone. In addition, the appearance features represented by the specific action may include, for example, highly visible safety clothing or safety vests that ensure visibility even in a dark environment by reflecting light, or not wearing protective equipment that is mandatory in a specific operation. The change in the position of the person includes, for example, the worker OP entering an area where an accident or injury may occur, such as an area where a worker OP is not expected to be outside the pedestrian passage or an area close to the operating equipment 2. An example of the method by which the action detection unit 12 detects a specific action will be specifically described. First, the action detection unit 12 detects the area in the captured image (image information IMI) where the worker OP is represented. Next, for the worker OP represented in the detected area, the action detection unit 12 may detect from the image information IMI that a specific action has occurred using a learned model that has been machine-learned so that posture estimation and skeleton estimation are possible. The action detection unit 12 transmits the information indicating the type of the detected specific action and the image information IMI in which the specific action is detected to the second storage unit 23. In addition, the action detection unit 12 transmits to the position estimation unit 13 and the individual identification unit 15 the image internal position information indicating the area in the image (image information IMI) captured by the imaging unit 3 where the worker OP in which the specific action is detected is represented, and the image information IMI in which the specific action is detected. Further, the action detection unit 12 transmits detection information indicating that a specific action has been detected to the time information acquisition unit 14 and the severity acquisition unit 17.

[0026] The position estimation unit 13 acquires the in-image position information and the image information IMI from the action detection unit 12. Based on the map information regarding the map of the area for detecting a specific action stored in the map information storage unit 22, the in-image position information, and the image information IMI, the position estimation unit 13 estimates the position on the map where the specific action has been detected. The map information may include, for example, information indicating the locations of places to put luggage and other items, the location of Facility 2, the locations of roads and pedestrian passages, the shapes of the walls and floors in the detection area, the coordinates of the detection area, and the floor number of the room where the detection area is located. The position estimation unit 13 transmits the position information, which is information regarding the estimated position, to the second storage unit 23. Note that the position information may include, for example, information such as the position where the specific action has been detected within the detection area, the coordinates of the detected position, and the floor number of the room where the detection area is located.

[0027] The time information acquisition unit 14 acquires detection information from the action detection unit 12. When the time information acquisition unit 14 acquires the detection information, it acquires time information indicating the current time. The time information acquisition unit 14 may acquire, as the current time, for example, the standard time in the area for detecting a specific action from a predetermined NTP (Network Time Protocol) server. The time information acquisition unit 14 transmits the time information to the second storage unit 23. Also, the time information acquisition unit 14 transmits the time information to the environmental information acquisition unit 16. Note that when a specific action continues to occur, such as a state where a prohibited action (e.g., walking while using a smartphone) continues, the time information acquisition unit 14 may acquire, as the time information, information indicating the duration from the start time when the specific action was detected to the end time when the detection ended.

[0028] The individual identification unit 15 acquires the in-image position information and the image information IMI from the behavior detection unit 12. Based on the individual identification information used to identify the operator OP, which is possessed by the individual identification information database 212, the in-image position information, and the image information IMI, the individual identification unit 15 performs face recognition on the operator OP represented in the area where the specific behavior is detected in the image (image information IMI) captured by the imaging unit 31 to identify the individual. The individual identification information may include, for example, the face image, name, gender, age, section, department, or company name to which the operator OP belongs for each operator OP. For example, the individual identification unit 15 acquires face information regarding the face of the operator OP represented in the area where the specific behavior is detected in the image (image information IMI) captured by the imaging unit 31. The face information may include, for example, the face image of the operator OP represented at the position where the specific behavior is detected in the image (image information IMI) captured by the imaging unit 31. Further, the face information may include information such as points that are facial features, such as the starting point of the eyebrows, the tip of the nose, and both ends of the mouth, the arrangement of the feature points, and the size of the face area. The individual identification unit 15 identifies the individual who performed the specific behavior by comparing the acquired face information with the individual identification information database 212. The individual identification unit 15 transmits the individual identification information of the identified operator OP to the second storage unit 23. Note that the individual identification unit 15 may use a learned model obtained by performing machine learning using individual identification information in advance so that the person represented in the image can be identified from the face of the operator OP represented in the area where the specific behavior is detected, and identify the person who performed the specific behavior.

[0029] The environmental information acquisition unit 16 acquires time information. The environmental information acquisition unit 16 transmits the acquired time information to the sensor management device 7. The sensor management device 7 acquires the time information and transmits the temperature information TI, the humidity information HI, and the illuminance information II at the time indicated by the time information to the environmental information acquisition unit 16. The environmental information acquisition unit 16 acquires the temperature information TI, the humidity information HI, and the illuminance information II at the time indicated by the time information from the sensor management device 7. The environmental information acquisition unit 16 transmits the temperature information TI, the humidity information HI, and the illuminance information II acquired from the sensor management device 7 to the second storage unit 23. Note that the environmental information acquisition unit 16 may continuously or at a predetermined time interval acquire the temperature information TI, the humidity information HI, and the illuminance information II from the sensor management device 7. In that case, the environmental information acquisition unit 16 transmits the temperature information TI, the humidity information HI, and the illuminance information II at the time indicated by the time information among the acquired information to the second storage unit 23.

[0030] In addition, in FIG. 2, an example is shown in which the environmental information acquisition unit 16 acquires temperature information TI, humidity information HI, and illuminance information II. However, the present embodiment is not limited to this example, and the environmental information acquisition unit 16 acquires arbitrary environmental information indicating the environment of the detection area from an environmental sensor. The environmental sensor is a sensor that measures the environment within the detection area, and may be, for example, the above-described temperature and humidity sensor 5, illuminance sensor 6, or the like. A plurality of such environmental sensors of the same type (for example, temperature and humidity sensors) may be installed at different positions, or a plurality of different types of sensors may be installed at the same position or different positions. The environmental information may include, for example, information about the environment such as weather, carbon dioxide concentration, volume, illuminance, wind speed, etc., or may include the degree of congestion indicating the population density in the detection area or a predetermined area within the detection area (for example, any area obtained by equally dividing the detection area). By the environmental information acquisition unit 16 acquiring various types of environmental information, the administrator who views the information aggregated by the information aggregation system 1 can accurately analyze the factors that induce specific actions. In addition, by the environmental information acquisition unit 16 acquiring environmental information that does not seem to be a factor that can induce specific actions, the administrator can newly discover the factors that induce specific actions and the factors that induce accidents. Note that when people are crowded, other people may get in the way, and people may fall without noticing a step, fall from a footrest because they cannot see their feet, or come into contact with an obstacle or a person. Therefore, the index of the degree of congestion can be a factor that induces specific actions. Also, in the dark or when it is too bright and dazzling, people may fall without noticing a step or an obstacle, or come into contact without noticing someone else. Therefore, the index of illuminance can be a factor that induces specific actions. Note that the factors that induce specific actions and the factors that induce accidents may be simply referred to as "factors" hereinafter.

[0031] Note that, in the present embodiment, an example is shown in which the environmental information acquisition unit 16 acquires environmental information from the sensor management device 7 included in the information aggregation system 1. However, the present embodiment is not limited to this example, and the information aggregation system 1 does not necessarily have to include the sensor management device 7. When the information aggregation system 1 does not include the sensor management device 7, the environmental information acquisition unit 16 acquires the environmental information acquired by the environmental sensors via the network NW.

[0032] The severity acquisition unit 17 acquires detection information from the behavior detection unit 12. The severity acquisition unit 17 transmits information indicating the severity of the specific behavior detected by the behavior detection unit 12 to the second storage unit 23. The severity is an index for classifying a specific behavior, a minor injury among the actually occurred accidents or injuries, and a serious injury among the actually occurred accidents or injuries. The severity may be defined, for example, as 1 for a specific behavior, 2 for a minor injury among the actually occurred accidents, and 3 for a serious injury among the actually occurred accidents. That is, the severity acquisition unit 17 automatically transmits the severity as 1 to the second storage unit 23 for the specific behavior detected by the behavior detection unit 12. Note that a minor injury may be, for example, one that has no impact on work even if first aid is provided for an injury or the like. A serious injury may be, for example, one that requires medical treatment and may affect work. Further, the severity may be defined by further subdividing the actually occurred accidents, or may be 1 for a specific behavior and 2 for the collectively actually occurred accidents.

[0033] In addition, the second storage unit 23 may store information on an actually-occurred accident transmitted from the terminal device 30 by an administrator's operation, in addition to information on the detected specific behavior. Information on an actually-occurred accident may include, for example, the type of accident, the position on the map where it occurred within the detection area, the time of occurrence, the personal identification information of the worker OP who suffered damage due to the occurrence of the accident, the environmental information at the time of the accident, the severity of the worker OP (e.g., severity is 2 or 3), and the like. By the second storage unit 23 storing information on an actually-occurred accident in addition to information on the detected specific behavior, the administrator can also analyze locations and situations where accidents are likely to develop from specific behaviors. For example, if a person falls in a room where the floor is slippery and is injured by a tool that has not been tidied up, even if there was a reminder to tidy up the tool immediately after use, the floor of the room remains slippery. Therefore, the worker OP may fall and be injured or fractured by something other than the tool. In addition to information on specific behavior, in the case of an actually-occurred accident, it can be analyzed that since the specific behavior of "falling" has occurred, the environment is slippery, and if there is a dangerous object such as a tool nearby, it has developed into an injury. This makes it possible to discover factors that induce accidents and specific behaviors and take countermeasures. Also, since it is difficult to determine the severity from the appearance of the worker OP, such as a minor fall (severity is 1) but actually having a fracture (severity is 2 or more), the administrator may operate the terminal device 30 to change the severity of the specific behavior with a severity of 1 stored in the second storage unit 23 to a value of 2 or more.

[0034] The first storage unit 21 includes a specific behavior information database 211, a personal identification information database 212, and a learned model information database 213. Note that the specific behavior information database 211, the personal identification information database 212, and the learned model information database 213 may be stored in different storage units from each other or any of them.

[0035] The specific action information database 211 stores specific action information. The terminal device 30 transmits the specific action information to the specific action information database 211 by an administrator's operation. The specific action information database 211 stores the specific action information transmitted from the terminal device 30.

[0036] The personal identification information database 212 stores personal identification information. The terminal device 30 transmits the personal identification information to the personal identification information database 212 by an administrator's operation. The personal identification information database 212 stores the personal identification information transmitted from the terminal device 30.

[0037] The learned model information database 213 stores learned models necessary for each functional unit of the information aggregating device 10 to realize its functions. The learned model information database 213 may store, for example, a learned model obtained by performing machine learning so that a specific action can be detected by estimating the skeleton or posture of a person performing a specific action represented in an image from the captured image. Further, the learned model information database 213 may store a learned model obtained by performing machine learning so that the position on the map where a specific action has occurred can be estimated within the detection area from the in-image position indicating the area where the specific action has been detected in the captured image and the map information of the captured area (detection area). Furthermore, the learned model information database 213 may store a learned model obtained by performing machine learning so that a person represented in the image can be identified from the personal identification information and the face of the operator OP represented in the area where the specific action has been detected.

[0038] The map information storage unit 22 stores map information. The terminal device 30 transmits the map information to the map information storage unit 22 by an administrator's operation. The map information storage unit 22 stores the map information transmitted from the terminal device 30.

[0039] The second storage unit 23 stores the information indicating the type of the detected specific action transmitted by the action detection unit 12, the image information IMI in which the specific action is detected, the position information transmitted by the position estimation unit 13, the time information transmitted by the time information acquisition unit 14, the personal identification information transmitted by the personal identification unit 15, the environmental information (temperature information TI, humidity information HI, and illuminance information II) transmitted by the environmental information acquisition unit 16, and the severity transmitted by the severity acquisition unit 17 in association with each other. Further, in the second storage unit 23, for the actually occurred accident transmitted by the terminal device 30, the type of the accident, the position on the map where the accident occurred within the detection area, the time of occurrence, the personal identification information of the worker OP who suffered damage due to the occurrence of the accident, the environmental information at the time when the accident occurred, and the severity of the worker OP are stored in association with each other. Note that the information stored in the second storage unit 23 in association may be hereinafter referred to as information regarding a specific action.

[0040] The information processing unit 18 acquires information about the evaluation items (search conditions) and display formats selected by the administrator from the terminal device 30. Based on the information about the evaluation items, the information processing unit 18 extracts from the second storage unit 23 the information regarding the detected specific action and the information selected as the evaluation items. Based on the extracted information, the information processing unit 18 generates information in the display format selected by the administrator. The information processing unit 18 transmits the generated information (hereinafter, may be referred to as generated information) to the output unit 19.

[0041] The output unit 19 acquires the generated information from the information processing unit 18. The output unit 19 transmits the acquired generated information to the terminal device 30. The terminal device 30 displays the generated information acquired from the output unit 19 to the administrator.

[0042] An example of the generated information displayed on the terminal device 30 will be described with reference to FIGS. 3 to 7.

[0043] FIG. 3 is a first diagram for explaining an example of the generated information displayed on the terminal device. With reference to this figure, a first display image IM1, which is an example of the configuration of the image displayed on the terminal device 30, will be described. The first display image IM1 includes a search condition selection unit 1001, a search button 1002, a search result count display unit 1003, a search result display unit 1004, a heat map display button 1005, a cross-tabulation display button 1006, a scatter plot display button 1007, and a log-off button 1008.

[0044] The administrator selects the conditions (search conditions) for extracting information from the information stored in the second storage unit 23. The search conditions selected by the administrator are displayed in the option column of the search condition selection unit 1001. Note that the administrator may input arbitrary conditions in the option column of the search condition selection unit 1001 without selecting the search conditions from the defined options. FIG. 3 shows an example in which, as the search conditions, the specific action of "falling / contacting", the period from August 1, 2023 to August 31, 2023, the time zone from 0:00 to 24:00 (all times), the location of "all" (the entire detection area), and the relevant persons of "all" are selected. After selecting the search conditions and pressing the search button 1002, the information processing unit 18 extracts the information that satisfies the search conditions selected from the information stored in the second storage unit 23. The number of information that satisfies the search conditions selected by the search condition selection unit 1001, that is, the number of search results, is displayed on the search result count display unit 1003. Note that information (items) stored in the second storage unit 23 other than "specific action", "period", "time zone", "location", and "relevant persons" may be displayed in the search condition selection unit 1001. For example, "illuminance", "temperature", "humidity", "crowdedness", "weather", "gender", "age", and "severity" may be displayed in the search condition selection unit 1001, or instead of the specific action, the actually occurred accidents may be displayed.

[0045] The information extracted (searched) under the selected search conditions is displayed on the search result display section 1004. FIG. 3 shows an example of the information extracted, where for each specific action, "date and time", "location (coordinates)", "illuminance", "temperature", "humidity", "crowdedness", "weather", "person involved", "gender", "age", "severity", and "video at the time when the specific action was performed" are respectively displayed. The administrator selects the display format of the extracted information by selecting the heat map display button 1005, the cross-tabulation display button 1006, or the scatter diagram display button 1007. In addition, when the administrator finishes using the system (hereinafter sometimes simply referred to as the system) that displays information on the terminal device 30, the administrator selects the logoff button 1008.

[0046] FIG. 4 is a second diagram for explaining an example of the generated information displayed on the terminal device. With reference to this figure, a second display image IM2, which is an example of the configuration of the image displayed on the terminal device 30 when the administrator selects the heat map display button 1005, will be described. The second display image IM2 includes a display item selection section 2001, a heat map display section 2002, and a back button 2003.

[0047] The administrator selects the item to be displayed on the heat map display section 2002 from the display item selection section 2001. FIG. 4 shows an example of selecting "fall" as the item to be displayed on the heat map display section 2002 among the specific actions that occurred in the time period from 8:00 to 12:00. Note that as the items of the information to be displayed, items other than "time period" and "detection result (type of specific action)", for example, environmental information, gender, age, etc. can be selected, or two or more can be selected, or the administrator can input them.

[0048] The heat map display unit 2002 displays the heat map of the item selected by the display item selection unit 2001. Fig. 3 shows an example of displaying a heat map that visualizes the positions and frequencies where falls occur on the map of the interior shown in Fig. 1. By using the heat map to display the extracted information, the administrator can visually understand the positions where specific actions are likely to occur, that is, the positions where factors exist. By being able to visually understand the positions where factors exist, the administrator can analyze and take countermeasures against the factors based on the situations around the positions where specific actions are likely to occur. For example, when there are many falls on the anti-slip mat using unevenness, it can be analyzed that the unevenness of the anti-slip mat may be worn down and instead become slippery. Accordingly, countermeasures such as replacing the anti-slip mat with a new one or giving a warning about the slippery condition can be taken. Also, for example, when there are many falls at a position with a small step, it can be analyzed that the operator OP is likely to overlook the step. Accordingly, countermeasures such as giving a warning, installing lighting, or installing a slope to eliminate the step can be taken to make the step easier to recognize. When the return button 2003 is selected by the administrator, the terminal device 30 displays the first display image IM1 again.

[0049] Fig. 5 is a third diagram for explaining an example of the generated information displayed on the terminal device. With reference to this figure, a third display image IM3, which is an example of the configuration of the image displayed on the terminal device 30 when the administrator selects the cross-tabulation table display button 1006, will be described. The third display image IM3 includes a cross-tabulation table display unit 3001, an Excel output button 3002, a return button 3003, and a graph display button 3004.

[0050] Among a plurality of types of specific actions, one or more types of specific actions selected by the administrator are defined as a first element group, and each of the one or more types of specific actions selected is defined as a first element. Specifically, in FIG. 5, falling and contact, which are the types of specific actions selected, are the first element group, and falling and contact are each a first element. Also, among the evaluation items selected by the administrator, items other than the types of specific actions are defined as a second element group, and each item divided under a predetermined condition is defined as a second element. Specifically, in FIG. 5, among the items selected by the administrator, time zone, illuminance, temperature, humidity, congestion, gender, age, and severity are collectively the second element group. Also, for example, "from 8:00 to 12:00", "from 12:00 to 16:00", "from 16:00 to 20:00", "from 20:00 to 24:00" in the time zone, and "from 0 to 500 [Lm]", "from 500 to 1000 [Lm]", "from 1000 to 1500 [Lm]", "from 1500 to 2000 [Lm]" in the illuminance are each a second element. The cross-tabulation display unit 3001 displays the number of any first element (for example, falling) that satisfies the condition (for example, from 8:00 to 12:00) indicated by any second element, with the first element and the second element being respectively associated. Also, the cross-tabulation display unit 3001 further displays the ratio of the number of any first element that satisfies the condition indicated by any second element to the sum of the numbers of each first element. By displaying the extracted information using the cross-tabulation, the administrator can analyze the factors that induce specific actions. In FIG. 5, for example, factors are analyzed based on the item of time zone. Among the specific actions that occurred, the falls that occurred from 8:00 to 12:00 and the falls that occurred from 16:00 to 20:00 each account for approximately 30% of the total falls and contacts that occurred. Therefore, for example, it can be analyzed that due to the concentration of people during commuting, there is a possibility of falling without noticing steps or obstacles at the feet. As a result, measures such as raising awareness of the existence of steps or obstacles can be taken.

[0051] By selecting the Excel output button 3002, the administrator can output the cross-tabulation table as an Excel (registered trademark) file. When the return button 3003 is selected by the administrator, the terminal device 30 displays the first display image IM1 again in the same manner as when the return button 2003 is selected.

[0052] By selecting the graph display button 3004, the administrator can display, for each first element, a graph showing, for each item in the second element group, the ratio of the number of first elements that satisfy the condition indicated by the second element to the number of first elements. With reference to FIG. 6, the displayed graph will be specifically described. FIG. 6 is a fourth diagram for explaining an example of the generated information displayed on the terminal device. With reference to this figure, a fourth display image IM4, which is an example of the configuration of the image displayed on the terminal device 30 when the administrator selects the graph display button 3004, will be described. The fourth display image IM4 includes a 100% stacked bar graph (inverted) 4001, a 100% stacked bar graph (contact) 4002, and a return button 4003.

[0053] The 100% stacked bar graph (fall) 4001 is a graph showing the percentage of the number of falls that occurred "from 8:00 to 12:00", "from 12:00 to 16:00", "from 16:00 to 20:00", and "from 20:00 to 24:00" with respect to the total number of falls. The 100% stacked bar graph (contact) 4002 is a bar graph showing the percentage of the number of contacts that occurred "from 8:00 to 12:00", "from 12:00 to 16:00", "from 16:00 to 20:00", and "from 20:00 to 24:00" with respect to the total number of contacts. By using the 100% stacked bar graph to display the percentage of the number of occurrences of a specific action for each condition, the administrator can more easily analyze the conditions under which the specific action is likely to occur, that is, the factors that induce the specific action, than when using numbers to display. When the return button 4003 is selected by the administrator, the terminal device 30 displays the third display image IM3 again. Although FIG. 6 shows an example in which a 100% stacked bar graph is displayed, the present embodiment is not limited to this example, and any graph that represents a ratio or value may be used. For example, a graph using a figure such as a band graph, a pie graph, an annular graph, or a bar graph may be used, or a graph representing a change such as a line graph may be used.

[0054] FIG. 7 is a fifth diagram for explaining an example of the generated information displayed on the terminal device. With reference to this figure, an example of the configuration of an image displayed on the terminal device 30 when the administrator selects the scatter plot display button 1007, that is, the fifth display image IM5, will be described. The fifth display image IM5 includes an axis item selection unit 5001, a scatter plot display unit 5002, an Excel output button 5003, and a return button 5004.

[0055] The manager selects or inputs the items to be displayed on the X-axis and Y-axis using the axis item selection unit 5001. The scatter diagram display unit 5002 displays a two-dimensional scatter diagram having the items selected or input in the axis item selection unit 5001 as axes. Fig. 7 shows an example in which "age" is selected as the X-axis (first axis) and "illuminance" is selected as the Y-axis (second axis), and points indicating "severity" are plotted. By displaying using the scatter diagram, the manager can analyze the factors that induce specific behaviors based on the correlation between the two items on the X-axis and Y-axis. For example, in Fig. 7, a specific behavior with a severity of 1 most frequently occurs regardless of age in an environment of 900 [Lm], which is suitable for performing detailed visual tasks such as sorting and inspection in a textile factory, typesetting and proofreading in a printing factory, and analysis in a chemical factory. Therefore, it can be analyzed that the factors inducing the specific behavior may be other than the relationship between age and illuminance (decrease in visual acuity due to aging). Also, when working in an environment of 400 [Lm] in the 50s, accidents with a severity of 2 and a severity of 3, that is, mild or severe damage, have occurred, but when working in an environment of 800 [Lm], the occurrence of specific behaviors with a severity of 1 has increased. Therefore, it can be analyzed that the increase in the occurrence of specific behaviors and accidents in the 50s may be due to the fact that it becomes easier to fall as one ages rather than due to illuminance. In Fig. 7, the age information selected as the first axis is the first information, and the illuminance information selected as the second axis is the second information. Note that the Y-axis may be used as the first axis and the X-axis may be used as the second axis.

[0056] The manager can output the scatter diagram as an Excel file by selecting the Excel output button 5003. When the return button 5004 is selected by the manager, the terminal device 30 displays the first display image IM1 again in the same manner as when the return button 2003 is selected.

[0057] In addition, in FIGS. 5 and 7, an example of outputting an Excel file is shown. However, the present embodiment is not limited to this example, and the cross-tabulation table or the information of the scatter diagram may be output in any data format such as a CSV (Comma Separated Value) file.

[0058] With reference to FIGS. 8 to 14, each device included in the information aggregation system 1 and the flow of processing between the devices will be described.

[0059] FIG. 8 is a flowchart for explaining an example of the processing flow of the information aggregation device. The information aggregation device 10 performs machine learning so as to be able to detect that a specific action has been performed by estimating at least one of the skeleton or the posture of the person performing the specific action represented in the captured image according to the instruction for learning the specific action transmitted from the terminal device 30 (step S101). The learned model is stored in the learned model information database 213.

[0060] The information aggregation device 10 stores (registers) the map information acquired from the terminal device 30 in the map information storage unit 22 (step S102). The information aggregation device 10 stores (registers) the personal identification information acquired from the terminal device 30 in the personal identification information database 212 (step S103).

[0061] The action detection unit 12 included in the information aggregation device 10 monitors whether a specific action has occurred based on the captured image (image information IMI). When the action detection unit 12 does not detect that a specific action has occurred from the image information IMI, that is, when it is determined that the specific action has not occurred (step S104; No), the monitoring of whether the specific action has occurred is continued. When the action detection unit 12 detects that a specific action has occurred from the image information IMI (step S104; Yes), the action detection unit 12 transmits the name of the type of the detected specific action to the second storage unit 23. Also, the position estimation unit 13 estimates the position on the map where the specific action has occurred based on the image information IMI and the map information. The position estimation unit 13 transmits the position information regarding the estimated position to the second storage unit 23. That is, the information aggregation device 10 stores the name of the type of the specific action and the position information regarding the estimated position in the second storage unit 23 (step S105).

[0062] The information aggregation device 10 acquires time information indicating the time when the specific action has occurred using the time information acquisition unit 14 and stores it in the second storage unit 23 (step S106). The information aggregation device 10 identifies the person who has performed the specific action using the person identification unit 15 and stores the personal identification information of the identified person in the second storage unit 23 (step S107). The information aggregation device 10 acquires the temperature information TI and the humidity information HI at the occurrence time using the temperature and humidity sensor 5 and stores them in the second storage unit 23 (step S108). The information aggregation device 10 acquires the illuminance information II at the occurrence time using the illuminance sensor 6 and stores it in the second storage unit 23 (step S109). That is, the information aggregation device 10 acquires the environmental information at the time when the specific action has occurred using the environmental sensor.

[0063] The information aggregation device 10 generates the information to be displayed on the terminal device 30 based on the information stored in the second storage unit 23 from step S105 to step S109 (step S110).

[0064] FIG. 9 is a flowchart for explaining the processing flow of the system for displaying the generated information on the terminal device. The administrator operates the terminal device 30 to log in to the system (step S201). When the login is successful, a home screen is displayed on the terminal device 30 (step S202). The administrator operates the terminal device 30 to select search conditions, that is, the conditions for extraction from the information stored in the second storage unit 23 (step S203). The administrator selects the display format for displaying the searched (extracted) information (step S204).

[0065] When the administrator selects the heat map (step S204; heat map), a heat map that visualizes the positions and frequencies where specific actions occur is displayed on the map registered in step S102 on the terminal device 30 (step S205). Note that the administrator can further select items to be displayed on the heat map by operating the terminal device 30 as needed.

[0066] When the administrator selects the cross-tabulation table (step S204; cross-tabulation table), the searched information is displayed on the terminal device 30 using the cross-tabulation table (step S206). The administrator operates the terminal device 30 to select the graph display button 3004 for any item displayed in the cross-tabulation table (step S207). A graph (for example, a 100% stacked bar graph) of the item selected by the administrator is displayed on the terminal device 30 (step S208). The terminal device 30 outputs an Excel file according to the administrator's operation (step S209). Note that the processing from step S207 to step S209 is not necessarily required.

[0067] When the administrator selects the scatter diagram (step S204; scatter diagram), the administrator further selects the items to be displayed on the X-axis and Y-axis respectively (step S210). A scatter diagram having the items selected by the administrator on the X-axis and Y-axis is displayed on the terminal device 30 (step S211). The terminal device 30 outputs an Excel file according to the administrator's operation (step S212). Note that the processing in step S212 is not necessarily required.

[0068] After the manager views the generated information displayed in any display format, the manager ends (logs off) the use of the system (step S213).

[0069] FIG. 10 is a sequence diagram for explaining an example of the flow of processing for learning a specific action. With reference to this figure, an example of the processing for learning a specific action (step S101) will be described. The terminal device 30 acquires authentication information for logging in by an operation of the manager (step S301). The terminal device 30 transmits the acquired authentication information to the information aggregation device 10 (step S302). The information aggregation device 10 performs an authentication process based on the authentication information acquired from the terminal device 30 (step S303). The information aggregation device 10 transmits display information including information on the menu screen to be displayed on the terminal device 30 to the terminal device 30 (step S304). The terminal device 30 acquires the display information from the information aggregation device 10 and displays the menu screen (step S305).

[0070] The terminal device 30 transmits (uploads) image information IMI about an image of a specific action to the information aggregation device 10 by an operation of the manager (step S306). The information aggregation device 10 transmits the acquired image information IMI to the specific action information database 211 (step S307). The specific action information database 211 stores the transmitted image information IMI. That is, the specific action information database 211 stores images (still images, moving images) uploaded by the terminal device 30 (step S308).

[0071] The terminal device 30 instructs, by an administrator's operation, the information aggregation device 10 to learn so as to be able to detect that a specific action has occurred from the captured image (step S309). The information aggregation device 10 performs machine learning according to the learning instruction (step S310). The information aggregation device 10 transmits the learned model to the learned model information database 213. The learned model information database 213 stores the transmitted learned model (step S311). Note that according to this learned model, the specific action instructed to learn can be detected from the captured image. Hereinafter, learning so as to be able to detect a specific action may be described as "registering a specific action".

[0072] The learned model information database 213 transmits completion information indicating that the registration of the specific action has been completed to the information aggregation device 10. When the information aggregation device 10 acquires the completion information, it transmits display information for displaying a result screen indicating that the registration of the specific action has been completed to the terminal device 30 (step S312). The terminal device 30 displays a result screen indicating that the registration of the specific action has been completed based on the display information transmitted from the information aggregation device 10 (step S313).

[0073] FIG. 11 is a sequence diagram for explaining an example of the flow of the process of registering map information. An example of the process of registering map information (step S102) will be described with reference to this figure. The processes performed in steps S401 to S405 are the same as the processes performed in steps S301 to S305.

[0074] The terminal device 30 transmits (uploads) the map information of the detection area (imageable area) to the information aggregation device 10 by an administrator's operation (step S406). The information aggregation device 10 transmits the acquired map information to the map information storage unit 22 (step S407). The map information storage unit 22 stores the transmitted map information. That is, the map information storage unit 22 stores the map information uploaded by the terminal device 30 (step S408).

[0075] The map information storage unit 22 transmits completion information indicating that the storage of the map information has been completed to the information aggregation device 10. When the information aggregation device 10 acquires the completion information, it transmits display information for displaying a result screen indicating that the storage of the map information has been completed to the terminal device 30 (step S409). The terminal device 30 displays a result screen indicating that the storage of the map information has been completed based on the display information transmitted from the information aggregation device 10 (step S410).

[0076] FIG. 12 is a sequence diagram for explaining an example of the flow of processing for registering personal identification information. An example of the processing for registering personal identification information (step S103) will be described with reference to this figure. The processing performed from step S501 to step S505 is the same as the processing performed from step S301 to step S305.

[0077] The terminal device 30 transmits (uploads) personal identification information used to identify a person (for example, worker OP) who may be in the detection area to the information aggregation device 10 by an administrator's operation (step S506). The information aggregation device 10 transmits the acquired personal identification information to the personal identification information database 212 (step S507). The personal identification information database 212 stores the transmitted personal identification information. That is, the personal identification information database 212 stores the personal identification information uploaded by the terminal device 30 (step S508).

[0078] The personal identification information database 212 transmits completion information indicating that the storage of the personal identification information has been completed to the information aggregation device 10. When the information aggregation device 10 acquires the completion information, it transmits display information for displaying a result screen indicating that the storage of the personal identification information has been completed to the terminal device 30 (step S509). The terminal device 30 displays a result screen indicating that the storage of the personal identification information has been completed based on the display information transmitted from the information aggregation device 10 (step S510).

[0079] FIG. 13 is a sequence diagram for explaining an example of the flow of processing for storing information regarding a detected specific action in the second storage unit. With reference to this figure, an example of the processing (from step S105 to step S109) for storing information regarding a detected specific action in the second storage unit 23 will be described. The video recorder 4 transmits the image information IMI being recorded to the information aggregator 10 (step S601). The information aggregator 10 monitors whether a specific action is occurring with respect to the transmitted image information IMI. When the information aggregator 10 detects that a specific action has occurred (step S602), it acquires position information regarding the position where the specific action has occurred (step S603). Also, the information aggregator 10 acquires time information regarding the time when the specific action has occurred, that is, the time when it detects that the specific action has occurred (step S604). The information aggregator 10 transmits the type of the detected specific action, the position information, and the time information to the second storage unit 23. The second storage unit 23 stores the transmitted type of the specific action, the position information, and the time information in association with each other (step S605).

[0080] The information aggregator 10 acquires face information from the image information IMI in which it has been detected that a specific action has occurred. The information aggregator 10 transmits the face information to the personal identification information database 212 (step S606). The personal identification information database 212 collates the transmitted face information with the personal identification information database 212 (step S607). The personal identification information database 212 transmits information regarding the collation result to the information aggregator 10. The information aggregator 10 identifies the person who has performed the specific action based on the transmitted collation result (step S608). The information aggregator 10 transmits the personal identification information of the identified worker OP to the second storage unit 23. The second storage unit 23 stores the transmitted personal identification information in further association (step S609).

[0081] The information aggregating device 10 sends time information to the sensor management device 7 and instructs it to send the environmental information acquired by each environmental sensor at the time when a specific action occurred (step S610). The sensor management device 7 sends the environmental information acquired by each environmental information at the time when the specific action occurred to the information aggregating device 10 based on the time information (step S611). The information aggregating device 10 sends the environmental information transmitted from the sensor management device 7 to the second storage unit 23 (step S612). The second storage unit 23 further stores the transmitted environmental information in an associated manner (step S613).

[0082] FIG. 14 is a sequence diagram for explaining the flow of processing for generating information to be displayed on the terminal device. With reference to the figure, an example of the processing (step S110) for generating information to be displayed on the terminal device 30 based on the information stored in the second storage unit 23 will be described. The processing performed in steps S701 to S705 is the same as the processing performed in steps S301 to S305.

[0083] The administrator inputs search conditions (evaluation items) by operating the terminal device 30. The terminal device 30 sends search condition information regarding the input search conditions to the information aggregating device 10 (step S706). The information aggregating device 10 searches for information that satisfies the search conditions from the information stored in the second storage unit 23 by sending the transmitted search condition information to the second storage unit 23 (step S707).

[0084] The second storage unit 23 sends the result of the search according to the transmitted search conditions, that is, the information that satisfies the search conditions, to the information aggregating device 10 (step S708). The information aggregating device 10 sends display information for displaying the search results to the terminal device 30 (step S709). The terminal device 30 displays the search results based on the display information transmitted from the information aggregating device 10 (step S710). [Embodiment 2]

[0085] FIG. 15 is a block diagram for explaining the configuration of the information aggregation system according to Embodiment 2. With reference to this figure, an example of the configuration of the information aggregation system 1A will be described. The information aggregation system 1A is similar to the information aggregation system 1 in that it detects the occurrence of a specific action and collects information regarding the detected specific action. The information aggregation system 1A is different from the information aggregation system 1 in that the imaging device 3A includes functional units similar to the action detection unit 12, the position estimation unit 13, and the individual identification unit 15 provided in the information aggregation device 10. The information aggregation system 1A includes the imaging device 3A, the temperature and humidity sensor 5, the illuminance sensor 6, the sensor management device 7, the information aggregation device 10A, and a plurality of terminal devices 30. Note that descriptions of matters explained in Embodiment 1 may be omitted.

[0086] The imaging device 3A has an action detection unit 42, a position estimation unit 43, and an individual identification unit 45. Note that the action detection unit 42, the position estimation unit 43, and the individual identification unit 45 each have functions similar to those of the action detection unit 12, the position estimation unit 13, and the individual identification unit 15, respectively. Note that each functional unit of the imaging device 3A may be realized using an electronic circuit as necessary. Further, each functional unit may not be included in a single device, and may be in a form that constitutes the imaging device 3A from a plurality of devices.

[0087] The imaging unit 31 images a predetermined forward angular field region. The imaging unit 31 transmits image information IMI regarding the captured image to the action detection unit 42.

[0088] The action detection unit 42 acquires image information IMI from the imaging unit 31. The action detection unit 42 refers to the specific action information database 211 to acquire specific action information. The action detection unit 42 detects (monitors) whether the image information IMI contains an image corresponding to a specific action. The action detection unit 42 detects that a specific action has occurred in the detection area based on the specific action information in the specific action information database 211 and the acquired image information IMI. The action detection unit 42 transmits information indicating the type of the detected specific action and the image information IMI in which the specific action is detected to the transmission unit 32. Also, similar to the action detection unit 12, the action detection unit 42 transmits the in-image position information and the image information IMI in which the specific action is detected to the position estimation unit 43 and the individual identification unit 45.

[0089] The position estimation unit 43 acquires the in-image position information and the image information IMI in which the specific action is detected from the action detection unit 42. Also, the position estimation unit 43 refers to the map information storage unit 22 to acquire map information. The position estimation unit 43 estimates the position on the map where the specific action is detected, similar to the position estimation unit 13. The position estimation unit 43 transmits the position information regarding the estimated position to the transmission unit 32.

[0090] The individual identification unit 45 acquires the in-image position information and the image information IMI from the action detection unit 42. The individual identification unit 45 acquires face information regarding the face of the worker OP represented in the area where the specific action is detected in the image (image information IMI) in which the specific action is detected, that is, the feature points of the face of the worker OP, etc. The individual identification unit 45 transmits the acquired face information to the transmission unit 32.

[0091] The transmission unit 32 acquires the information indicating the type of the detected specific action and the image information IMI in which the specific action is detected transmitted by the action detection unit 42, the position information transmitted by the position estimation unit 43, and the face information transmitted by the individual identification unit 45. The transmission unit 32 transmits the information acquired from the action detection unit 42, the position estimation unit 43, and the individual identification unit 45, respectively, to the reception unit 51 in an associated manner.

[0092] The information aggregation device 10A is similar to the information aggregation device 10 in that it includes a time information acquisition unit 14, an environmental information acquisition unit 16, a severity acquisition unit 17, an information processing unit 18, an output unit 19, a first storage unit 21, a map information storage unit 22, and a second storage unit 23. The information aggregation device 10A is different from the information aggregation device 10 in that it does not include an image acquisition unit 11, an action detection unit 12, a position estimation unit 13, and an individual identification unit 15. Further, the information aggregation device 10A is different from the information aggregation device 10 in that it further includes a reception unit 51 and an individual verification unit 55.

[0093] The reception unit 51 acquires, from the transmission unit 32, information in which information indicating the type of the detected specific action, image information IMI in which the specific action is detected, position information, time information, and face information are associated with each other. The reception unit 51 transmits the associated information to the second storage unit 23. Further, the reception unit 51 transmits detection information indicating that the specific action has been detected to the time information acquisition unit 14 and the severity acquisition unit 17. Note that the reception unit 51 does not necessarily transmit the face information to the second storage unit 23.

[0094] The individual verification unit 55 acquires face information from the reception unit 51. The individual verification unit 55 identifies the individual who has performed the specific action by comparing the acquired face information with the individual identification information database 212. The individual verification unit 55 transmits the individual identification information of the identified worker OP to the second storage unit 23. The second storage unit 23 acquires information from the reception unit 51, the time information acquisition unit 14, the individual verification unit 55, the environmental information acquisition unit 16, and the severity acquisition unit 17, and stores the same information as the second storage unit 23 according to the first embodiment.

[0095] Note that FIGS. 2 and 5 show an example in which one temperature and humidity sensor 5 and one illuminance sensor 6 are provided respectively. However, the present embodiment is not limited to this example, and a plurality of environmental sensors may be provided respectively. By providing a plurality of environmental sensors respectively, that is, by installing environmental sensors at a plurality of points within the detection area, the administrator can accurately know the environment in which the specific action has occurred.

[0096] In addition, in Embodiment 1 and Embodiment 2, an example of detecting specific actions performed by the worker OP is shown. However, this embodiment is not limited to this example, and it may be a person in the area to be detected, for example, a person who does not perform work such as a site supervisor, or an employee.

[0097] The information aggregation system according to this embodiment can register actions that may lead to unexpected accidents or injuries but do not result in industrial accidents as specific actions, and thus can take countermeasures in advance before an industrial accident occurs. Therefore, safety can be further enhanced. Also, for example, in the late-night hours, according to an information aggregation system that registers actions that may lead to deterioration of public order, such as standing still in front of a building for a long time, squatting down, or gathering in a large group, as specific actions, security companies and the police can efficiently patrol (make rounds). Also, according to the information aggregation system, actions that may lead to deterioration of public order, that is, an environment where squatting down or gathering in a large group is likely, can be analyzed and countermeasures can be taken.

[0098] [Summary of this Embodiment] According to the above-described embodiment, the information aggregation system includes a first storage unit 21 that stores specific action information including the type of a specific action that is an action to be detected among the attention actions to be taken in advance, and information indicating the appearance characteristics represented by the specific action and the change in the position where a person is located; a map information storage unit 22 that stores map information of an area (detection area) for detecting the specific action; an image acquisition unit 11 that acquires image information IMI regarding an image of the detection area; an action detection unit 12 that detects the specific action from the image information IMI acquired by the image acquisition unit 11 based on the specific action information; a position estimation unit 13 that estimates the position where the detected specific action has occurred in the detection area indicated by the map information based on the image information IMI acquired by the image acquisition unit 11; a second storage unit 23 that stores in association information regarding the type of the specific action detected by the action detection unit 12 and position information regarding the position estimated by the position estimation unit 13; and an output unit 19 that outputs a result based on the information stored in the second storage unit 23. In many cases, when a worker OP who has fallen (performed a specific action) is not injured and has not damaged the facility 2, the worker continues the work without reporting to anyone. If a report is not made by the worker OP who has performed the specific action, it is impossible to take countermeasures when there are factors that induce the specific action or an accident. According to the information aggregation system according to the present embodiment, even in an event where no report is made because an accident does not occur, by recording, it is possible to discover factors that induce the specific action or an accident and take countermeasures. Therefore, according to the information aggregation system, safety can be improved.

[0099] Also, according to the above-described embodiment, in the information aggregation system, the first storage unit 21 further stores personal identification information used to identify a person who may be in the detection area, and based on the personal identification information, from the image information IMI acquired by the image acquisition unit 11, it further includes a personal identification unit 15 that identifies the person who has performed the detected specific action. The second storage unit 23 further stores, in association, the personal identification information about the person identified by the personal identification unit 15. According to the information aggregation system according to this embodiment, by identifying the worker OP who has performed the specific action, it is possible to instruct the worker OP who is performing the prohibited action not to perform the prohibited action. In addition, an administrator who browses information using the information aggregation system can analyze the factors that induce the specific action by using the gender, age, etc. of the worker OP.

[0100] Also, according to the above-described embodiment, the information aggregation system further includes a time information acquisition unit 14 that acquires time information regarding the time when the detected specific action occurred, and the second storage unit 23 further stores the time information in association. By the information aggregation system according to this embodiment storing the information regarding time in the second storage unit 23 in association, the administrator can analyze the factors that induce the specific action by using the information about the frequency and time zone when the specific action occurs.

[0101] Also, according to the above-described embodiment, the information aggregation system further includes an environmental information acquisition unit 16 that acquires environmental information regarding the environment of the space where the detected specific action occurred, and the second storage unit 23 further stores the environmental information in association. By the information aggregation system according to this embodiment storing the environmental information in the second storage unit 23 in association, the administrator can analyze the factors that induce the specific action from various viewpoints. In addition, by the information aggregation system acquiring environmental information that does not seem to be a factor that can induce the specific action, the administrator can newly discover the factors that induce the specific action and the factors that induce accidents.

[0102] Also, according to the above-described embodiment, the information aggregation system further includes a severity acquisition unit 17 that acquires severity, which is an index for classifying specific actions, minor accidents among the actually occurred accidents, and serious accidents among the actually occurred accidents, and the second storage unit 23 stores the severity in further association. According to the information aggregation system according to the present embodiment, by the second storage unit 23 storing information related to the actually occurred accidents in addition to the information related to the detected specific actions, the administrator can analyze the places and situations where it is easy for the specific actions to develop into accidents based on the severity caused by the detected specific actions and the actually occurred accidents.

[0103] Also, according to the above-described embodiment, the information aggregation system further includes an information processing unit 18 that performs a process of extracting information related to the type of the detected specific action and information related to the item to be evaluated from the information stored in the second storage unit 23, and the output unit 19 outputs the information processed by the information processing unit 18. According to the information aggregation system according to the present embodiment, since the information stored in the second storage unit 23 can be extracted (searched) under arbitrary conditions, when analyzing the factors that induce specific actions, the administrator can easily narrow down the factors. Also, based on the number of search results under the selected conditions, the factors that induce a certain specific action can be analyzed.

[0104] Also, according to the above-described embodiment, in the information aggregation system, the information processing unit 18 further performs a process of generating information indicating the number of times of specific actions for each position in the detection area based on the information extracted including the information related to the type of the detected specific action and the position information related to the estimated position. By displaying the extracted information using a heat map, the administrator can visually understand the positions where specific actions are likely to occur, that is, the positions where factors exist. According to the information aggregation system according to the present embodiment, since the administrator can visually understand the positions where factors exist, based on the situations around the positions where specific actions are likely to occur, the factors can be analyzed and countermeasures can be taken.

[0105] Also, according to the above-described embodiment, in the information aggregation system, the information processing unit 18 is two-dimensional information. On the first axis, it has information regarding the type of the detected specific action among the extracted information. As the second axis orthogonal to the first axis, it has one or more types of information different from the information regarding the type of the detected specific action. Any one of the plurality of elements of the information on the first axis (the first element) is associated with any one of the plurality of elements of each of the information included in the second axis (the second element), and information indicating the number of events satisfying the two associated elements is generated. According to the cross-tabulation table, based on the ratios and values shown in the table, conditions under which a specific action is likely to occur (for example, falls are likely to occur from 8:00 to 12:00) can be analyzed. Therefore, according to the information aggregation system according to the present embodiment, the administrator can analyze the factors inducing the specific action by using the cross-tabulation table to display the extracted information.

[0106] Also, according to the above-described embodiment, in the information aggregation system, the information processing unit 18, based on a first element group that is a plurality of elements of the information on the first axis and a second element group that is a plurality of elements of any of the information on the second axis, uses a graph to generate information indicating the ratio of each element of the other element group in at least any one of the elements of one of the first element group or the second element group. According to the information aggregation system according to the present embodiment, by visualizing using a graph the ratios and frequencies at which various types of specific actions occur under a certain condition, the burden on the administrator to analyze the factors inducing the specific action can be reduced.

[0107] Also, according to the above-described embodiment, in the information aggregation system, the information processing unit 18 generates information that is two-dimensional information, having first information that is any of the extracted information on the first axis and second information that is different in type from the first information among the extracted information on the second axis orthogonal to the first axis. According to the scatter plot displayed by the information aggregation system according to the present embodiment, the administrator can analyze the factors inducing the specific action based on the correlation between the two items on the X-axis (the first axis) and the Y-axis (the second axis).

[0108] Also, according to the above-described embodiment, in the information aggregation system, the attention actions include actions that may lead to crime or deterioration of public order. For example, according to an information aggregation system in which actions that may lead to deterioration of public order, such as standing still in front of a building for a long time, sitting down, or gathering in a large group, are registered as specific actions during late-night hours, security companies and the police can patrol efficiently. Further, according to the information aggregation system, it is possible to analyze and take countermeasures against actions that may lead to deterioration of public order, that is, environments where it is easy to sit down or gather in a large group.

[0109] Note that each device included in the information aggregation system 1 and the information aggregation system 1A in the above-described embodiment, and the entire or part of the functions of the functional parts of each device may be realized by recording a program for realizing these functions on a computer-readable recording medium, reading the program recorded on this recording medium into a computer system, and executing it. Here, the "computer system" is assumed to include hardware such as an OS and peripheral devices.

[0110] Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, etc., and a recording part such as a hard disk built in a computer system. Furthermore, the "computer-readable recording medium" also includes something that dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and something that holds a program for a certain time, like a volatile memory inside a computer system that becomes a server or a client in that case. Also, the above program may be for realizing a part of the above-described functions, and may further be realized in combination with a program already recorded in the computer system for realizing the above-described functions.

[0111] The above has described in detail an embodiment of the present invention with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the gist of the present invention. Also, it is possible to combine the configurations described in each of the above embodiments and each example.

Explanation of Reference Signs

[0112] 1…Information aggregation system, 10…Information aggregation device, 11…Image acquisition unit, 12…Action detection unit, 13…Position estimation unit, 14…Time information acquisition unit, 15…Individual identification unit, 16…Environmental information acquisition unit, 17…Severity acquisition unit, 18…Information processing unit, 19…Output unit, 21…First storage unit, 211…Specific action information database, 212…Individual identification information database, 213…Learned model information database, 22…Map information storage unit, 23…Second storage unit, 1A…Information aggregation system, 3…Imaging device, 31…Imaging unit, 32…Transmission unit, 3A…Imaging device, 4…Recorder, 5…Temperature and humidity sensor, 6…Illuminance sensor, 7…Sensor management device, 30…Terminal device

Claims

1. A first storage unit that stores specific action information including the type of a specific action, which is an action to be detected among attention actions for which countermeasures are to be taken in advance, and information indicating the appearance characteristics and the change in the position of a person represented by the specific action; A map information storage unit that stores map information of an area for detecting the specific action; An image acquisition unit that acquires an image of the area to be detected; An action detection unit that detects the specific action from the image acquired by the image acquisition unit based on the specific action information; A position estimation unit that estimates the position where the detected specific action occurred in the area to be detected indicated by the map information based on the image acquired by the image acquisition unit; A second storage unit that stores in association information regarding the type of the specific action detected by the action detection unit and the position estimated by the position estimation unit; An output unit that outputs a result based on the information stored in the second storage unit; An information aggregation system comprising the above.

2. The first storage unit further stores personal identification information used for identifying a person who may be in the area to be detected, further comprises a personal identification unit that identifies the person who performed the detected specific action from the image acquired by the image acquisition unit based on the personal identification information, and the second storage unit further stores in association the personal identification information about the person identified by the personal identification unit. The information aggregation system according to Claim 1.

3. further comprises a time information acquisition unit that acquires time information regarding the time when the detected specific action occurred, and the second storage unit further stores the time information in association. The information aggregation system according to Claim 1.

4. further comprises an environmental information acquisition unit that acquires environmental information regarding the environment of the space where the detected specific action occurred, and the second storage unit further stores the environmental information in association. The information aggregation system according to Claim 1.

5. further comprises a severity acquisition unit that acquires a severity, which is an index for classifying the specific action, minor damage among the actually occurred accidents, and severe damage among the actually occurred accidents, and the second storage unit further stores the severity in association. The information aggregation system according to Claim 1.

6. The information processing apparatus further includes an information processing unit that performs a process of extracting, from the information stored in the second storage unit, information related to the type of the detected specific action and information related to an item to be evaluated. The output unit outputs the information processed by the information processing unit. The information aggregation system according to any one of claims 1 to 5.

7. The information processing unit further performs a process of generating information indicating the number of times of the specific action for each position in the detection area based on the information extracted including the information related to the type of the detected specific action and the information related to the estimated position. The information aggregation system according to claim 6.

8. The information processing unit is two-dimensional information, has, on a first axis, information related to the type of the detected specific action among the extracted information, and has, as a second axis orthogonal to the first axis, one or more pieces of information different in type from the information related to the type of the detected specific action, and any one of the plurality of elements of the information on the first axis is associated with any one of the plurality of elements of each of the information included in the second axis, and generates information indicating the number of events satisfying the two associated elements. The information aggregation system according to claim 6.

9. Based on a first element group that is a plurality of elements of the information on the first axis and a second element group that is a plurality of elements of any of the information on the second axis, the information processing unit uses a figure to determine at least one of the elements of one of the first element group or the second element group, and generates information indicating the ratio of each element of the other element group. The information aggregation system according to claim 8.

10. The information processing unit generates information that is two-dimensional information, has, on a first axis, first information that is any of the extracted information, and has, on a second axis orthogonal to the first axis, second information different in type from the first information among the extracted information. The information aggregation system according to claim 6.

11. The cautionary actions include actions that may lead to crime or deterioration of public order. The information aggregation system according to claim 6.

12. An image acquisition step of acquiring an image of an area in which a specific action, which is a detected action among cautionary actions for which countermeasures are to be taken in advance, is detected. An action detection step of detecting the specific action from the image acquired in the image acquisition step based on the specific action information including the type of the specific action and information indicating the appearance features represented by the specific action and the change in the position where a person is located; A position estimation step of estimating the position where the detected specific action has occurred in the detection area indicated in the map information of the area for detecting the specific action based on the image acquired in the image acquisition step; An output step of outputting a result based on information associating the information on the type of the specific action detected in the action detection step with the position estimated in the position estimation step; An information aggregation method having the above.

Images