Management device, management method, and program

Abstract

A management device (10) includes a motion detection unit (11), a related image identification unit (12), a determination unit (13), and an output unit (14). The motion detection unit (11) detects a predetermined motion performed by a person from image data on an image obtained by capturing a certain location including the person. The related image identification unit (12) identifies a predetermined related image that is related to the safety of the person. The determination unit (13) determines whether the person is in a safe situation on the basis of the detected motion and a positional relationship between the person performing the motion and the related image. The output unit (14) outputs determination information including a result of the determination made by the determination unit (13).

Description

[Technical Field]

[0001] The present disclosure relates to a management device, a management method, and a computer-readable medium. [Background technology]

[0002] 2. Description of the Related Art Various techniques have been developed to ensure the safety of workers in designated spaces such as construction sites.

[0003] For example, Patent Document 1 discloses a technique for obtaining work site measurement data by separating the movements of workers and movements of non-workers at a work site, and then determining risk.

[0004] Patent Document 2 discloses a technology that acquires the operating status of equipment, detects the position and orientation of a worker, and determines that a predetermined combination of the operating status of the equipment and the position and orientation of the worker is an inappropriate state.

[0005] Patent Document 3 discloses a technology that acquires identification information of each of multiple workers who are simultaneously present in a danger zone, and executes a safety operation when at least one of the multiple workers enters a detection zone set for at least one of the workers. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2019-101549 [Patent Document 2] Japanese Patent Application Laid-Open No. 2019-191748 [Patent Document 3] Japanese Patent Application Publication No. 2018-202531 Summary of the Invention [Problem to be solved by the invention]

[0007] However, because workers on-site perform a variety of actions, it is difficult to integrate various perspectives and manage safety comprehensively. In addition, simpler technology is needed to keep workers safe.

[0008] In view of the above-mentioned problems, an object of the present disclosure is to provide a management device etc. that can efficiently and easily manage the safety of workers. [Means for solving the problem]

[0009] A management device according to one aspect of the present disclosure includes a motion detection means, a related image identification means, a determination means, and an output means. The motion detection means detects a predetermined motion performed by a person from an image captured of a predetermined location including the person. The related image identification means identifies a related image showing a predetermined object or area related to the person's safety from image data of the image captured of the predetermined location. The determination means determines whether the person is in a safe situation based on the detected motion and the positional relationship between the person performing the motion and the object or area shown in the related image. The output means outputs determination information including the result of the determination made by the determination means.

[0010] In a management method according to one aspect of the present disclosure, a computer executes the following processes: the computer detects a predetermined action being performed by a person from image data of an image captured of a predetermined location including the person; the computer identifies a related image showing a predetermined object or area related to the safety of the person from the image captured of the predetermined location; the computer determines whether the person is in a safe situation based on the detected action and the positional relationship between the person performing the action and the object or area shown in the related image; and the computer outputs determination information including the result of the determination.

[0011] A computer-readable medium according to one aspect of the present disclosure stores a program that causes a computer to execute the following management method. The computer detects a predetermined action being performed by a person from an image captured of a predetermined location including the person. The computer identifies a related image showing a predetermined object or area related to the person's safety from image data of the image captured of the predetermined location. The computer determines whether the person is in a safe situation based on the detected action and the positional relationship between the person performing the action and the object or area shown in the related image. The computer outputs determination information including the result of the determination. [Effects of the Invention]

[0012] The present disclosure makes it possible to provide a management device or the like that can efficiently and easily manage the safety of workers. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 2 is a block diagram of a management device according to the first embodiment. [Figure 2] 1 is a flowchart showing a management method according to the first embodiment. [Figure 3] FIG. 10 is a diagram illustrating the overall configuration of a management system according to a second embodiment. [Figure 4] FIG. 10 is a diagram showing skeletal data extracted from image data. [Figure 5] FIG. 10 is a diagram for explaining a registered action database according to the second embodiment. [Figure 6] FIG. 10 is a diagram for explaining a first example of a registration operation according to the second embodiment. [Figure 7] FIG. 10 is a diagram for explaining a second example of the registration operation according to the second embodiment. [Figure 8] FIG. 10 is a diagram for explaining a safety standard database according to the second embodiment. [Figure 9] FIG. 2 is a diagram showing a first example of an image captured by a camera. [Figure 10] FIG. 10 is a diagram showing skeleton data extracted by the management device. [Figure 11]FIG. 10 is a diagram showing a related image identified by the management device. [Figure 12] This is a diagram in which skeletal data and related images are superimposed on an image captured by a camera. [Figure 13] FIG. 10 is a diagram showing a second example of an image captured by the camera. [Figure 14] FIG. 10 is a diagram showing a third example of an image captured by the camera. [Figure 15] FIG. 10 is a diagram showing a fourth example of an image captured by the camera. [Figure 16] FIG. 10 is a diagram showing the overall configuration of a management system according to a third embodiment. [Figure 17] FIG. 10 is a block diagram of an authentication device according to a third embodiment. [Figure 18] 10 is a flowchart showing a management method according to a third embodiment. [Figure 19] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0014] The present disclosure will be described below through embodiments, but the disclosure according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations are omitted as necessary.

[0015] <Embodiment 1> First, a first embodiment of the present disclosure will be described. Fig. 1 is a block diagram of a management device 10 according to the first embodiment. The management device 10 shown in Fig. 1 analyzes the posture and movement of a person included in an image captured by a camera installed at a predetermined work site, for example, and manages whether the person is performing work etc. in accordance with predetermined safety standards.

[0016] The management device 10 mainly comprises a motion detection unit 11, a related image identification unit 12, a determination unit 13, and an output unit 14. In this disclosure, "posture" refers to the shape of at least a part of the body, and "motion" refers to the state of assuming a predetermined posture over time. "Movement" is not limited to cases where posture changes, but also includes cases where a constant posture is maintained. Therefore, simply referring to "motion" may also include posture.

[0017] The motion detection unit 11 detects a predetermined motion performed by a person from image data of an image captured at a predetermined location including the person. The image data is image data of a plurality of consecutive frames capturing a person performing a series of motions. The image data is, for example, image data in a predetermined format such as H.264 or H.265. That is, the image data may be a still image or a video.

[0018] The predetermined movement detected by the movement detection unit 11 is estimated from, for example, an image of a person's body extracted from image data. The movement detection unit 11 detects from the image of the person's body that the person is performing a predetermined task. The predetermined task is, for example, a predetermined pattern of work that is likely to be performed at a work site.

[0019] The related image identifying unit 12 identifies a predetermined related image related to the safety of a person. The predetermined related image is a predetermined image and may include, for example, a helmet, gloves, safety shoes, and belt worn by a worker. The predetermined related image may also be an image related to tools and heavy machinery used by a worker. The predetermined related image may also be an image related to facilities, passageways, and predetermined areas used by a worker. The related image identifying unit 12 may identify a related image by recognizing the above-mentioned images from an image captured by a camera. The related image identifying unit 12 may also identify a predefined area superimposed on an image captured by a camera.

[0020] The determination unit 13 determines whether a person included in an image captured by the camera is in a safe situation. When making this determination, the determination unit 13 refers to the motion detected by the motion detection unit 11. When making this determination, the determination unit 13 also calculates or refers to the positional relationship between the person performing the detected motion and the related image identified by the related image identification unit 12.

[0021] The positional relationship may be, for example, the distance between the person involved in the detected action and the related image. The positional relationship may also indicate, for example, whether the related image is located at a predetermined position on the body of the person involved in the detected action. The positional relationship may also indicate whether the person involved in the detected action is included in the related image as a predetermined region.

[0022] The determination unit 13 may calculate or refer to the positional relationship by analyzing the angle of view, angle, etc. of the image from a predetermined object or scene included in the image captured by the camera. In this case, the positional relationship may correspond to the actual three-dimensional space of the captured image. The positional relationship may be calculated by estimating a pseudo three-dimensional space in the captured image. The positional relationship may be a positional relationship on a plane of the captured image. The determination unit 13 may calculate or refer to the above-mentioned positional relationship by previously setting the angle of view, angle, etc. of the image captured by the camera.

[0023] The output unit 14 outputs determination information including the result of the determination made by the determination unit 13. In this case, the determination information may indicate that the person whose movement is detected is safe, or may indicate that the person whose movement is detected is unsafe or in danger. The output unit 14 may output the above-mentioned determination information to, for example, a display device (not shown) included in the management device 10. The output unit 14 may also output the above-mentioned determination information to an external device communicatively connected to the management device 10.

[0024] Next, the processing executed by the management device 10 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the management method according to the first embodiment. The flowchart shown in Fig. 2 starts, for example, when the management device 10 acquires image data.

[0025] First, the action detection unit 11 detects a predetermined action being performed by a person from image data of an image captured of a predetermined location including the person (step S11). When the action detection unit 11 detects the predetermined action being performed by the person, it supplies information about the detected action to the determination unit 13.

[0026] Next, the related image specifying unit 12 specifies a predetermined related image related to the safety of the person (step S12). The related image specifying unit 12 supplies information on the specified related image to the determining unit 13.

[0027] Next, the determination unit 13 determines whether the person is safe or not based on the detected movement and the positional relationship between the person performing this movement and the related image (step S13). When the determination unit 13 generates determination information including the determination result, the generated determination information is supplied to the output unit 14.

[0028] Next, the output unit 14 outputs the determination information including the result of the determination to a predetermined output destination (step S14). When the output unit 14 outputs the determination information, the management device 10 ends the series of processes.

[0029] In the above process, step S11 and step S12 may be performed in the reverse order, may be performed simultaneously, or may be performed in parallel.

[0030] Although the first embodiment has been described above, the configuration of the management device 10 is not limited to the above. For example, the management device 10 may have a processor and a storage device, which are not shown in the figure. The storage device may include a storage device including a nonvolatile memory such as a flash memory or an SSD (Solid State Drive). In this case, the storage device of the management device 10 stores a computer program (hereinafter simply referred to as a program) for executing the above-described management method. The processor also loads the computer program from the storage device into a buffer memory such as a DRAM (Dynamic Random Access Memory) and executes the program.

[0031] Each component of the management device 10 may be implemented using dedicated hardware. Furthermore, some or all of the components may be implemented using general-purpose or dedicated circuits, processors, or a combination thereof. These may be implemented using a single chip, or multiple chips connected via a bus. Some or all of the components of each device may be implemented using a combination of the above-described circuits and programs. Furthermore, processors such as CPUs (Central Processing Units), GPUs (Graphics Processing Units), and FPGAs (Field-Programmable Gate Arrays) may be used. The description of the configurations described herein may also be applied to other devices or systems described below in this disclosure.

[0032] Furthermore, when some or all of the components of the management device 10 are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally or distributed. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, or the like, in which they are connected via a communication network. Furthermore, the functions of the management device 10 may be provided in a SaaS (Software as a Service) format. Furthermore, the above-described method may be stored on a computer-readable medium to cause a computer to execute the above-described method.

[0033] As described above, according to this embodiment, it is possible to provide a management device etc. that can efficiently and simply manage the safety of workers.

[0034] <Embodiment 2> Next, a second embodiment of the present disclosure will be described. Fig. 3 is a diagram showing the overall configuration of a management system 2 according to the second embodiment. The management system 2 includes a management device 20 and a camera 100. The management device 20 and the camera 100 are connected to each other via a network N1 so as to be able to communicate with each other.

[0035] The camera 100 may also be referred to as an imaging device. The camera 100 includes an objective lens and an image sensor, and captures images of the work site where it is installed at predetermined intervals. At the work site captured by the camera 100, for example, a person P10 who is a worker is present. The camera 100 captures at least a part of the body of the person P10 by capturing images of the work site.

[0036] Camera 100 generates image data for each captured image and sequentially supplies the data to management device 20 via network N1. The predetermined period may be, for example, 1 / 15th of a second, 1 / 30th of a second, or 1 / 60th of a second. Camera 100 may have functions such as panning, tilting, and zooming.

[0037] The management device 20 is a computer device with a communication function, such as a personal computer, a tablet PC, a smartphone, etc. The management device 20 includes an image data acquisition unit 201, a display unit 202, an operation reception unit 203, and a storage unit 210 in addition to the configuration described in the first embodiment.

[0038] In this embodiment, the motion detection unit 11 extracts skeletal data from image data. More specifically, the motion detection unit 11 detects an image area (body area) of a person's body from a frame image included in the image data and extracts (e.g., cuts out) it as a body image. Then, the motion detection unit 11 uses a skeletal estimation technique using machine learning to extract skeletal data of at least a part of the person's body based on features such as the person's joints recognized in the body image. The skeletal data is information including "key points" that are characteristic points such as joints, and "bone links" that indicate links between the key points. The motion detection unit 11 may use a skeletal estimation technique such as OpenPose. Note that in this disclosure, the above-mentioned bone links may also be simply referred to as "bones." Bones refer to pseudo-skeleton structures.

[0039] Furthermore, the movement detection unit 11 detects a predetermined posture or movement from the extracted person's skeletal data, and compares the skeletal data related to the retrieved registered movement with the extracted person's skeletal data. When detecting a posture or movement, the movement detection unit 11 searches for registered movements registered in a registered movement database stored in the storage unit 210. Then, if the person's skeletal data and the skeletal data related to the registered movement are similar, the movement detection unit 11 recognizes this skeletal data as a predetermined posture or movement. In other words, if the movement detection unit 11 detects a registered movement similar to the person's skeletal data, it associates the movement related to this skeletal data with the registered movement and recognizes it as a predetermined posture or movement. In other words, the movement detection unit 11 recognizes the type of person's movement by associating the person's skeletal data with the registered movement.

[0040] In the above-described similarity determination, the motion detection unit 11 detects posture or motion by calculating the degree of similarity between the shapes of elements constituting the skeletal data. Skeletal data has pseudo joint points or skeletal structures set as its components to indicate the posture of the body. The shape of the elements constituting the skeletal data can be, for example, the relative geometric relationship of the positions, distances, angles, etc. of other key points or bones when a certain key point or bone is used as a reference. Alternatively, the shape of the elements constituting the skeletal data can be, for example, the shape formed by a single integrated form of multiple key points or bones.

[0041] The movement detection unit 11 analyzes whether the relative shapes of the components between the two pieces of skeletal data being compared are similar. At this time, the movement detection unit 11 calculates the similarity between the two pieces of skeletal data. When calculating the similarity, the movement detection unit 11 may calculate the similarity using, for example, feature amounts calculated from the components of the skeletal data.

[0042] In addition, instead of the above similarity, the object of calculation by the movement detection unit 11 may be the similarity between a portion of the extracted skeletal data and skeletal data related to the registered movement, or the similarity between the extracted skeletal data and a portion of the skeletal data related to the registered movement, or the similarity between a portion of the extracted skeletal data and a portion of the skeletal data related to the registered movement.

[0043] The motion detection unit 11 may calculate the above-mentioned similarity directly or indirectly using the skeletal data. For example, the motion detection unit 11 may convert at least a part of the skeletal data into another format and calculate the above-mentioned similarity using the converted data. In this case, the similarity may be the similarity between the converted data itself, or may be a value calculated using the similarity between the converted data.

[0044] The conversion method may be normalization of the image size of the skeletal data, or conversion into a feature using the angle of the skeletal structure (i.e., the degree of bending of the joints), or the conversion method may be a 3D posture converted by a pre-trained machine learning model.

[0045] With the above-described configuration, the motion detection unit 11 in this embodiment detects motions similar to predetermined registered motions. The predetermined registered motions are, for example, information about typical work motions performed by a person at a work site. If the detected motion is similar to the predetermined registered motion, the motion detection unit 11 supplies a signal indicating that this motion is similar to the registered motion to the determination unit 13.

[0046] As described above, the movement detection unit 11 in this embodiment detects movements from skeletal data relating to the body structure of a person extracted from image data relating to an image including a person. That is, the movement detection unit 11 extracts an image of the body of person P10 from the image data and estimates a pseudo skeleton relating to the body structure of the extracted person. Furthermore, in this case, the movement detection unit 11 detects movements by comparing the skeletal data relating to the movements with skeletal data as registered movements based on the shapes of the elements constituting the skeletal data.

[0047] The movement detection unit 11 may detect posture or movement from skeletal data extracted from a single image data. The movement detection unit 11 may also detect movement from posture changes extracted in time series from multiple sets of image data captured at multiple different times. That is, the movement detection unit 11 detects posture changes of the person P10 from multiple frames. With this configuration, the management device 20 can flexibly analyze movements according to the state of changes in posture or movement to be detected. In this case, the movement detection unit 11 may also use a registered movement database.

[0048] The related image identification unit 12 in this embodiment identifies a predetermined object worn on a person's body as a related image. The predetermined object worn on a person's body is, for example, a helmet or a safety belt worn by a worker.

[0049] In this case, the determination unit 13 treats the detected movement as one element in the determination. The determination unit 13 also treats the positional relationship between the person performing this movement and the identified predetermined object as one element in the determination. For example, the determination unit 13 determines that it is unsafe if the position of the object does not correspond to the predetermined position of the person performing the predetermined movement. More specifically, for example, if the determination unit 13 detects that a person P10 performing predetermined civil engineering work at a work site is wearing a helmet on his / her head, it determines that the person P10 is safe. On the other hand, if the determination unit 13 does not detect that a person P10 performing predetermined civil engineering work at a work site is wearing a helmet on his / her head, it determines that the person P10 is unsafe (i.e., dangerous).

[0050] The related image identification unit 12 identifies, as a related image, an object having a predetermined danger area. Examples of objects having a predetermined danger area include heavy machinery such as trucks, cranes, and wheel loaders, and equipment such as cutters, concrete mixers, and high-voltage power supplies. A predetermined danger area can be set for these objects. For example, the danger area is a place where entry by anyone other than people performing a specific task is prohibited.

[0051] In this case, if there is a person in the danger zone who is performing an action different from the predetermined permitted action, the determination unit 13 determines that it is unsafe. More specifically, for example, if there is a person performing civil engineering work unrelated to the heavy machinery in the danger zone around the heavy machinery, the determination unit 13 determines that the person P10 is unsafe.

[0052] The related image identification unit 12 may identify a predetermined judgment area as the related image. The predetermined area is, for example, an area where a safety confirmation action is performed. In this case, the determination unit 13 determines whether the person is safe or not based on the positional relationship between the person performing the action and the judgment area. More specifically, for example, if the worker P10 performs a specified confirmation action in a judgment area where a safety confirmation action is required, the determination unit 13 determines that the person is safe. On the other hand, if the worker P10 does not perform the specified confirmation action in the judgment area, the determination unit 13 determines that the person is not safe.

[0053] The determination unit 13 in this embodiment determines whether or not a person is safe by referring to predetermined safety standard data. The determination unit 13 reads a safety standard database stored in the storage unit 210. The safety standard database includes multiple pieces of safety standard data. The safety standard data is data used when determining whether or not a person is safe, and includes data on the person's movements, data on related images, and data on the positional relationship between the person and the related images. The output unit 14 in this embodiment outputs the determination information generated by the determination unit 13 to the display unit 202.

[0054] The image data acquisition unit 201 is an interface that acquires image data supplied from the camera 100. The image data acquired by the image data acquisition unit 201 includes images captured by the camera 100 at predetermined intervals. The image data acquisition unit 201 supplies the acquired image data to the action detection unit 11 and the related image identification unit 12.

[0055] The display unit 202 is a display including a liquid crystal panel or an organic electroluminescence device. The display unit 202 displays the determination information output by the output unit 14, and presents the determination result to the user of the management device 20.

[0056] The operation reception unit 203 includes information input means such as a keyboard or a touchpad, and receives operations from a user who operates the management device 20. The operation reception unit 203 may be a touch panel that is superimposed on the display unit 202 and configured to operate in conjunction with the display unit 202.

[0057] The storage unit 210 is a storage means including a nonvolatile memory such as a flash memory. The storage unit 210 stores at least a registered action database and a safety standard database. The registered action database includes skeletal data as registered actions. The safety standard database includes multiple pieces of safety standard data. In other words, the storage unit 210 stores at least safety standard data regarding the positional relationship between a person involved in an action and an associated image.

[0058] Next, an example of detecting a person's posture will be described with reference to Fig. 4. Fig. 4 is a diagram showing skeletal data extracted from image data. The image shown in Fig. 4 is a body image F10 obtained by extracting the body of person P10 from an image captured by camera 100. In management device 20, movement detection unit 11 cuts out body image F10 from the image captured by camera 100, and further sets the skeletal structure.

[0059] The movement detection unit 11 extracts, for example, feature points that can be key points of the person P10 from the image. The movement detection unit 11 then detects key points from the extracted feature points. When detecting key points, the movement detection unit 11 refers to, for example, information learned by machine learning about the image of the key points.

[0060] In the example shown in Figure 4, the movement detection unit 11 detects the following key points of person P10: head A1, neck A2, right shoulder A31, left shoulder A32, right elbow A41, left elbow A42, right hand A51, left hand A52, right hip A61, left hip A62, right knee A71, left knee A72, right foot A81, and left foot A82.

[0061] Furthermore, the movement detection unit 11 sets bones connecting these key points as a pseudo-skeletal structure of person P10, as shown below: Bone B1 connects the head A1 and neck A2. Bone B21 connects the neck A2 and right shoulder A31, and bone B22 connects the neck A2 and left shoulder A32. Bone B31 connects the right shoulder A31 and right elbow A41, and bone B32 connects the left shoulder A32 and left elbow A42. Bone B41 connects the right elbow A41 and right hand A51, and bone B42 connects the left elbow A42 and left hand A52. Bone B51 connects the neck A2 and right hip A61, and bone B52 connects the neck A2 and left hip A62. Bone B61 connects the right hip A61 to the right knee A71, and bone B62 connects the left hip A62 to the left knee A72. Bone B71 connects the right knee A71 to the right foot A81, and bone B72 connects the left knee A72 to the left foot A82. After generating skeletal data related to the above-mentioned skeletal structure, the movement detection unit 11 uses the generated skeletal data to compare with registered movements.

[0062] Next, an example of the registered motion database will be described with reference to Fig. 5. Fig. 5 is a diagram for explaining the registered motion database according to the second embodiment. In the table shown in Fig. 5, registered motion IDs (identification, identifier) ​​are associated with a plurality of motion patterns. The motion pattern for a motion with a registered motion ID (or motion ID) of "R01" is "task M11". Similarly, the motion pattern for a registered motion ID of "R02" is "task M12", and the motion pattern for a registered motion ID of "R03" is "task M13". In addition to the predetermined tasks, the registered motion database may also have motion patterns for crouching and lying down as motion patterns for detecting dangerous situations.

[0063] As described above, the data on registered movements contained in the registered movement database is stored with each movement linked to a movement ID and a movement pattern. Each movement pattern is linked to one or more pieces of skeleton data. For example, a registered movement with a movement ID of "R01" includes skeleton data indicating a movement performed during a specific civil engineering task.

[0064] Skeleton data related to a registered motion will be described with reference to FIG. 6. FIG. 6 is a diagram for explaining a first example of a registered motion according to the second embodiment. FIG. 6 shows skeleton data related to a motion with a motion ID of "R01" among the registered motions included in the registered motion database. FIG. 6 shows a plurality of pieces of skeleton data including skeleton data F11 and skeleton data F12 arranged in the left-right direction. Skeleton data F11 is located to the left of skeleton data F12. Skeleton data F11 is a posture capturing a scene of a person performing a series of civil engineering work. Skeleton data F12 is a scene of a person performing a series of civil engineering work, and is a different posture from that of skeletal data F11.

[0065] 6 indicates that in the registered action with action ID "R01", the person takes the posture corresponding to the skeleton data F11 and then the posture corresponding to the skeleton data F12. Note that although two skeleton data have been described here, the registered action with action ID "R01" may include skeleton data other than the above-mentioned skeleton data.

[0066] Fig. 7 is a diagram for explaining a second example of a registered motion according to the second embodiment. Fig. 7 shows skeleton data F31 related to the motion with motion ID "R03" shown in Fig. 5. For the registered motion with motion ID "R03", only one skeleton data F31 indicating a person performing a guiding motion at a work site is registered.

[0067] As described above, the registered motions included in the registered motion database may include only one piece of skeletal data, or may include two or more pieces of skeletal data. The motion detection unit 11 compares the registered motions including the skeletal data with the skeletal data estimated from the image received from the image data acquisition unit 201, and determines whether there are any similar registered motions.

[0068] Next, the safety standard database will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining the safety standard database according to the second embodiment. The table shown in Fig. 8 shows the safety standard database, and is arranged in the left-right direction so that "action pattern," "related image," "positional relationship," and "determination" correspond to each other.

[0069] For example, in the top row of the table, "Task M11" is shown as the action pattern, and in the same row, "Image P11" is shown as the related image, "Image P11 on head A1" is shown as the positional relationship, and the judgment is "Safe." In this example, image P11 represents a helmet. In other words, the safety standard data shown here states that when a person is performing a specified civil engineering task (task M11), if the helmet (image P11) corresponds to the person's head (A1), it is "safe."

[0070] Similarly, the second row of the table shown in FIG. 8 lists "Task M11" as the action pattern, "Image P12" as the related image, "The distance between the worker and image P12 is less than Dth" as the positional relationship, and the judgment is "Danger." In this example, image P12 represents a truck. In other words, the safety standard data shown here states that when a person is performing a specified civil engineering task (task M11), if the distance between the person and the truck (image P12) is less than the threshold distance Dth, it is "Danger."

[0071] Similarly, the third row of the table shown in FIG. 8 lists "Task M13" as the action pattern, "Image P12" as the related image, "Skeletal data exists in the attention area of ​​image P12" as the positional relationship, and the judgment as "safe." In this example, it is assumed that an attention area corresponding to image P12 has been set. The safety standard data shown here is that when a person is performing a predetermined guiding action (Task M13), if the person (skeletal data) exists in the attention area of ​​the truck (image P12), it is "safe."

[0072] The safety standards database has been described above. The determination unit 13 of the management device 20 determines whether or not a person is safe by referring to the safety standards described above.

[0073] Next, the safety standard data will be explained while explaining specific examples of images. Fig. 9 is a diagram showing a first example of an image captured by a camera. Image F21 shown in Fig. 8 is an image captured by camera 100 and includes worker P10. Worker P10 is performing a predetermined civil engineering task at a work site. Management device 20 receives image data of this image and determines whether worker P10 is safe.

[0074] FIG. 10 is a diagram showing skeletal data extracted by the management device. Image F22 shown in FIG. 10 is a body image of person P10 extracted by the movement detection unit 11 and skeletal data generated by inferring from this body image. The skeletal data includes head A1. The movement detection unit 11 compares the skeletal data shown here with the registered movement database. Here, the skeletal data shown in FIG. 10 corresponds to task M11 of movement pattern R01. Furthermore, the movement detection unit 11 acquires skeletal data of a movement corresponding to task M11 at a different time after the image shown in FIG. 10. Therefore, the movement detection unit 11 determines that person P10 is performing task M11.

[0075] Fig. 11 is a diagram showing related images identified by the management device. Fig. 11 shows a state in which an image P11 of a helmet worn by a person P10 is detected in an image F21. The related image identification unit 12 can search for related images and detect the related image P11 by performing a predetermined convolution process on the image F21 using a known method such as HOG (Histogram of oriented gradients) or machine learning.

[0076] Fig. 12 is a diagram in which skeletal data and related images are superimposed on an image captured by a camera. The determination unit 13 of the management device 20 refers to the skeletal data shown in Fig. 10 and the related image shown in Fig. 11, and recognizes the positional relationship between them. As shown in Fig. 12, person P10, who is performing an action corresponding to task M11 with a registered action ID R01, has related image P11 (helmet) on his head A1. Therefore, the determination unit 13 determines that person P10 included in image F21 is safe.

[0077] Next, a further example of safety standard data will be described with reference to Fig. 13. Fig. 13 is a diagram showing a second example of an image captured by a camera. Image F23 shown in Fig. 13 is an image captured by camera 100, and includes a person P10 performing a task M11, which is a predetermined civil engineering task, and a related image P12 of a truck approaching person P10. The image shown here corresponds to the safety standard data shown in the second row of Fig. 8.

[0078] In image F23 shown in FIG. 13, the action detection unit 11 detects that person P10 is performing task M11 of action pattern R01. Furthermore, the related image identification unit 12 detects related image P12, which is a truck. Furthermore, the determination unit 13 calculates the distance D10 between person P10 and related image P12. In this example, the determination unit 13 calculates the distance between person P10 and the truck from a line connecting a point at the bottom center of the image of the identified person and a point at the bottom center of the image of the truck. At this time, the determination unit 13 is configured to be able to calculate the distance between any two points based on the camera's angle of view and shooting angle. Therefore, the determination unit 13 can determine whether the distance D10 is less than a predetermined threshold Dth. Therefore, if the distance D10 in image F23 is less than the threshold Dth, the determination unit 13 determines that the image is "dangerous," and if the distance D10 is equal to or greater than the threshold Dth, the determination unit 13 does not determine that the image is "dangerous."

[0079] In this way, the management device 20 determines whether a person is safe by referring to the person's movements and the positional relationship between the person and the related image. This allows the management device 20 to appropriately determine whether a situation is safe depending on the work that the person is doing.

[0080] The safety standard data will be further described with reference to Fig. 14. Fig. 14 is a diagram showing a third example of an image captured by a camera. Image F24 shown in Fig. 14 differs from Fig. 13 in the behavior of person P10. Person P10 in image F24 is performing task M13, which is the action of guiding a truck.

[0081] In image F24 shown in Fig. 14, the action detection unit 11 detects that person P10 is performing task M13 of action pattern R03. The related image identification unit 12 detects related image P12, which is a truck. The determination unit 13 then calculates the distance D10 between person P10 and related image P12. Because the action of person P10 in image F24 is not task M11, the determination unit 13 does not determine that person P10 is "dangerous."

[0082] In this way, by referring to the person's actions and the positional relationship between the person and the related image, the management device 20 may not determine that a situation is dangerous depending on the person's actions, even if the person and an object related to the related image are nearby. This allows the management device 20 to appropriately determine a dangerous situation depending on the person's work.

[0083] Fig. 15 is a diagram showing a fourth example of an image captured by a camera. Image F25 shown in Fig. 15 is an example of the safety standard data shown in the third row of the table shown in Fig. 8. In the example shown in image F24, a caution area corresponding to related image P12 is set.

[0084] In image F25 shown in FIG. 15, the action detection unit 11 detects that person P10 is performing task M13 of action pattern R03. Furthermore, the related image identification unit 12 detects related image P12, which is a truck. Furthermore, the determination unit 13 refers to the positional relationship between the attention area T10 linked to the related image P12 and person P10. Person P10 present in attention area T10 is performing task M13 as an action pattern. The safety standard database indicates that when a person is performing a predetermined guiding action (task M13), it is "safe" if the person (skeleton data) is present in the attention area of ​​the truck (image P12). Therefore, the determination unit 13 determines person P10 to be "safe."

[0085] In this way, the management device 20 can determine that only a person performing a predetermined action in a predetermined area is safe. Conversely, the management device 20 does not determine that a person performing an action other than a predetermined action in a predetermined area is safe. In other words, the management device 20 can determine that such a person is dangerous. With this configuration, the management device 20 can appropriately determine whether a person is in a safe or dangerous situation depending on the work that the person is doing and their positional relationship with the related image.

[0086] Although the configuration of the second embodiment has been described above, the management system 2 according to the second embodiment is not limited to the above configuration. For example, the number of cameras 100 included in the management system 2 is not limited to one, but may be multiple. Some of the functions of the motion detection unit 11 may be included in the camera 100. In this case, for example, the camera 100 may extract a body image of a person by processing the captured image. Alternatively, the camera 100 may further extract skeletal data of at least a part of the person's body from the body image based on features of the person's joints and the like recognized in the body image.

[0087] The management device 20 and the camera 100 may be able to communicate directly without going through the network N1. The management device 20 may include the camera 100. In other words, the management system 2 may be synonymous with the management device 20.

[0088] The motion detection unit 11 may detect the motions of multiple people from image data of an image captured of a location including multiple people. In this case, the determination unit 13 determines whether the people are safe or not based on the positional relationships between the multiple people and the associated images.

[0089] With the configuration described above, according to the second embodiment, it is possible to provide a management device and the like that can efficiently and simply manage the safety of workers.

[0090] <Embodiment 3> Next, a third embodiment will be described with reference to FIG. 16. FIG. 16 is a diagram showing the overall configuration of a management system 3 according to the third embodiment. The management system 3 shown in FIG. 16 includes a management device 30, a camera 100, an authentication device 300, and a management terminal 400. These components are connected to each other so as to be able to communicate with each other via a network N1. That is, the management system 3 according to this embodiment differs from the second embodiment in that it includes a management device 30 instead of the management device 20, and in that it includes an authentication device 300 and a management terminal 400.

[0091] The management device 30 identifies a predetermined person in cooperation with the authentication device 300, determines whether the identified person is safe or not, and outputs the determination result to the management terminal 400. The management device 30 differs from the management device 20 according to the second embodiment in that it has a person identification unit 15. Also, the management device 30 differs from the management device 20 according to the second embodiment in that the storage unit 210 of the management device 30 stores a person attribute database related to the person to be identified.

[0092] The person identification unit 15 identifies a person included in the image data. The person identification unit 15 identifies a person included in the image captured by the camera 100 by linking the authentication data of the person authenticated by the authentication device 300 with the attribute data stored in the person attribute database.

[0093] In this case, the output unit 14 outputs whether the identified person is safe or not to the management terminal 400. If the identified person is not safe, the output unit 14 outputs a warning signal corresponding to the identified person to the management terminal 400. That is, the output unit 14 in this embodiment outputs a predetermined warning signal when it is determined that the person is not safe.

[0094] The determination unit 13 may have a plurality of safety levels for determining whether a person is safe or not. In this case, the output unit 14 outputs a warning signal according to the safety level. With this configuration, the management device 30 can more flexibly manage safety.

[0095] The person attribute database stored in the storage unit 210 includes attribute data of the identified person. The attribute data includes the person's name, unique identifier, etc. The attribute data may also include data related to the person's work. That is, the attribute data may include, for example, the group to which the person belongs or the type of work the person performs. The attribute data may also include, for example, the person's blood type, age, or gender, which is safety-related data.

[0096] In this embodiment, the motion detection unit 11, the related image identification unit 12, and the determination unit 13 may make a determination based on the attribute data of the person. That is, for example, the motion detection unit 11 may check registered motions corresponding to the identified person. The related image identification unit 12 may recognize related images corresponding to the identified person. Furthermore, the determination unit 13 may make a determination by referring to safety standard data corresponding to the identified person. With this configuration, the management device 30 can make a determination customized for the identified person.

[0097] The authentication device 300 is a computer or server device including one or more arithmetic devices. The authentication device 300 authenticates people present at the work site from images captured by the camera 100 and supplies the authentication results to the management device 30. If the person authentication is successful, the authentication device 300 supplies the management device 30 with authentication data linked to the person attribute data stored in the management device 30.

[0098] The management terminal 400 is a dedicated terminal device having a tablet terminal, a smartphone, or a display device, and can receive the determination information generated by the management device 30 and present the received determination information to the manager P20. By recognizing the determination information presented on the management terminal 400 at the work site, the manager P20 can know whether the worker P10 is safe or not.

[0099] Next, the configuration of authentication device 300 will be described in detail with reference to Fig. 17. Fig. 17 is a block diagram of authentication device 300. Authentication device 300 authenticates a person by extracting a predetermined feature image from an image captured by camera 100. The feature image is, for example, a facial image. Authentication device 300 has an authentication storage unit 310, a feature image extraction unit 320, a feature point extraction unit 330, a registration unit 340, and an authentication unit 350.

[0100] The authentication storage unit 310 stores a person ID and the feature data of this person in association with each other. The feature image extraction unit 320 detects feature areas included in the image acquired by the camera 100 and outputs the detected feature areas to the feature point extraction unit 330. The feature point extraction unit 330 extracts feature points from the feature areas detected by the feature image extraction unit 320 and outputs data related to the feature points to the registration unit 340. The data related to the feature points is a collection of the extracted feature points.

[0101] The registration unit 340 issues a new person ID when registering the feature data. The registration unit 340 associates the issued person ID with the feature data extracted from the registered image and registers them in the authentication storage unit 310. The authentication unit 350 compares the feature data extracted from the feature image with the feature data in the authentication storage unit 310. If the feature data match, the authentication unit 350 determines that authentication is successful, and if the feature data do not match, the authentication unit 350 determines that authentication is unsuccessful. The authentication unit 350 notifies the management device 30 of the success or failure of authentication. Furthermore, if the authentication is successful, the authentication unit 350 identifies the person ID associated with the successful feature data and notifies the management device 30 of the authentication result including the identified person ID.

[0102] The authentication device 300 may authenticate a person using a means other than the camera 100. The authentication may be biometric authentication, or authentication using a mobile terminal, an IC card, or the like.

[0103] The processing performed by the management device 30 in this embodiment will be described with reference to Fig. 18. Fig. 18 is a flowchart showing a management method according to embodiment 3. The flowchart shown in Fig. 18 differs from the flowchart shown in Fig. 2 in the processing after step S13.

[0104] After step S13, the person identification unit 15 identifies the person related to the determination information from the image data and authentication data (step S21). Next, the output unit 14 outputs the determination information for the identified person to the management terminal 400 (step S22). After outputting the determination information to the management terminal 400, the management device 30 ends the series of processes.

[0105] The method executed by the management device 30 is not limited to the method shown in Fig. 18. The management device 30 may execute step S21 before step S13. Furthermore, the processes from step S11 to step S13 may be performed according to the person identified as described above.

[0106] With the above-described configuration, according to the third embodiment, it is possible to provide a management device and the like that can efficiently and simply manage the safety of workers.

[0107] <Example of hardware configuration> Hereinafter, a case will be described in which each functional configuration of the determination device according to the present disclosure is realized by a combination of hardware and software.

[0108] FIG. 19 is a block diagram illustrating an example of a hardware configuration of a computer. The management device of the present disclosure can realize the above-described functions by a computer 500 including the hardware configuration shown in the figure. The computer 500 may be a portable computer such as a smartphone or tablet terminal, or a stationary computer such as a PC. The computer 500 may be a dedicated computer designed to realize each device, or may be a general-purpose computer. The computer 500 can realize desired functions by installing a predetermined program.

[0109] The computer 500 has a bus 502, a processor 504, a memory 506, a storage device 508, an input / output interface 510 (an interface is also called an I / F (Interface)), and a network interface 512. The bus 502 is a data transmission path through which the processor 504, the memory 506, the storage device 508, the input / output interface 510, and the network interface 512 transmit and receive data to and from each other. However, the method of connecting the processor 504 and other components to each other is not limited to a bus connection.

[0110] The processor 504 is one of various processors such as a CPU, a GPU, an FPGA, etc. The memory 506 is a main storage device realized using a RAM (Random Access Memory) or the like.

[0111] The storage device 508 is an auxiliary storage device realized using a hard disk, an SSD, a memory card, a ROM (Read Only Memory), or the like. The storage device 508 stores programs for realizing desired functions. The processor 504 reads the programs into the memory 506 and executes them to realize the respective functional components of each device.

[0112] The input / output interface 510 is an interface for connecting the computer 500 with input / output devices. For example, the input / output interface 510 is connected to an input device such as a keyboard and an output device such as a display device.

[0113] The network interface 512 is an interface for connecting the computer 500 to a network.

[0114] Although an example of a hardware configuration in the present disclosure has been described above, the above-described embodiment is not limited to this. Any processing in the present disclosure can also be realized by causing a processor to execute a computer program.

[0115] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0116] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

[0117] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Appendix 1) a motion detection means for detecting a predetermined motion being performed by a person from an image captured of a predetermined location including the person; a related image identifying means for identifying a related image showing a predetermined object or area related to the safety of the person from images taken of the predetermined location; a determining means for determining whether the person is in a safe situation based on the detected movement and a positional relationship between the person performing the movement and an object or area shown in the related image; an output means for outputting determination information including the result of the determination made by the determination means; A management device comprising: (Appendix 2) the motion detection means detects the motion similar to a predetermined registered motion; 2. The management device of claim 1. (Appendix 3) the motion detection means detects the motion from skeletal data relating to the body structure of the person extracted from an image including the person; 3. The management device of claim 2. (Appendix 4) the motion detection means detects the motion by comparing the skeletal data relating to the motion with the skeletal data as the registered motion based on the form of the elements constituting the skeletal data; 4. The management device of claim 3. (Appendix 5) the motion detection means detects the type of the motion based on the registered motion; the determining means determines whether the person is in a safe situation based on the type of the action and the positional relationship between the person and the object or area indicated by the related image; 5. The management device according to any one of Supplementary notes 1 to 4. (Appendix 6) the movement detection means detects the movement from posture changes extracted in time series from each of a plurality of images taken at a plurality of different times; 6. The management device according to any one of Supplementary notes 1 to 5. (Appendix 7) a storage means for storing safety standard data relating to a positional relationship between the person and the related image involved in the action; The determination means determines whether or not the product is safe by referring to the safety standard data. 7. The management device according to any one of Supplementary notes 1 to 6. (Appendix 8) the related image specifying means specifies a predetermined object worn on the body of the person as the related image; The determination means determines that it is unsafe when the position of the object does not correspond to a predetermined position of the person performing the predetermined action. 8. The management device according to any one of Supplementary notes 1 to 7. (Appendix 9) the related image specifying means specifies an object having a predetermined dangerous area as the related image; The determination means determines that the danger area is unsafe when the person is performing an action other than the predetermined action permitted in the danger area. 8. The management device according to any one of Supplementary notes 1 to 7. (Appendix 10) The related image specifying means specifies a predetermined determination area as the related image, the determination means determines whether the person is safe or not based on a positional relationship between the person involved in the movement and the determination area. 8. The management device according to any one of Supplementary notes 1 to 7. (Appendix 11) the motion detection means detects the motions of the plurality of people from an image captured of a location including the plurality of people; the determining means determines whether the person is safe or not based on the positional relationship between each of the plurality of people and the related image. The management device according to any one of Supplementary Notes 1 to 10. (Appendix 12) the output means outputs a predetermined warning signal when it is determined that the person is unsafe. 12. The management device according to any one of Supplementary Notes 1 to 11. (Appendix 13) the determination means has a plurality of safety levels for determining whether the person is safe or not, The output means outputs the warning signal according to the safety level. 13. The management device of claim 12. (Appendix 14) Further, a person identification means for identifying the person included in the image is provided, the output means outputs the warning signal corresponding to the identified person when the identified person is unsafe. 14. The management device of claim 12 or 13. (Appendix 15) The computer Detecting a predetermined action being performed by a person from an image captured of a predetermined location including the person; identifying predetermined relevant images related to the safety of said person; determining whether the person is in a safe situation based on the detected movement and a positional relationship between the person performing the movement and the related image; outputting determination information including the result of the determination; Management method. (Appendix 16) Detecting a predetermined action being performed by a person from an image captured of a predetermined location including the person; identifying predetermined relevant images related to the safety of said person; determining whether the person is in a safe situation based on the detected movement and a positional relationship between the person performing the movement and the related image; outputting determination information including the result of the determination; A non-transitory computer-readable medium storing a program that causes a computer to execute a management method. [Explanation of symbols]

[0118] 2 Management System 3 Management System 10 Management device 11 Motion detection unit 12 Related image identification section 13 Judgment section 14 Output section 15 Person identification section 100 cameras 20 Management device 30 Management device 201 Image data acquisition unit 202 Display section 203 Operation reception section 210 Storage section 300 Authentication Device 310 Authentication storage unit 320 Feature Image Extraction Unit 330 Feature Point Extraction Unit 340 Registration Department 350 Authentication Department 400 Management terminal 500 computers 504 processor 506 memory 508 Storage Devices 510 Input / Output Interface 512 network interface N1 Network

Claims

1. A motion detection means for detecting a predetermined motion of a person from an image captured at a predetermined location; a related image identifying means for identifying a related image in the image that shows a predetermined object related to the safety of the person; a determination means for determining that the person performing the action is in a safe situation when the action in the attention area associated with the related image is a default action, and determining that the person performing the action is not in a safe situation when the action in the attention area is not a default action; an output means for outputting determination information including the result of the determination made by the determination means; A management device comprising:

2. The management device according to claim 1 , wherein the action detection means detects the action similar to a predetermined registered action.

3. 3. The management device according to claim 2, wherein the movement detection means detects the movement from skeletal data relating to the body structure of the person extracted from an image including the person.

4. The management device according to claim 3 , wherein the motion detection means detects the motion by comparing the skeletal data relating to the motion with the skeletal data as the registered motion based on the form of the elements constituting the skeletal data.

5. the motion detection means detects the type of the motion based on the registered motion; the determining means determines whether the person is in a safe situation based on the type of the action and the positional relationship between the person and the object or area indicated by the related image; The management device according to any one of claims 2 to 4.

6. The management device according to any one of claims 1 to 5, wherein the movement detection means detects the movement from posture changes extracted in time series from each of a plurality of images taken at a plurality of different times.

7. a storage means for storing safety standard data relating to a positional relationship between the person and the related image involved in the action; The determination means determines whether or not the product is safe by referring to the safety standard data. The management device according to any one of claims 1 to 6.

8. the related image specifying means specifies a predetermined object worn on the body of the person as the related image; The determination means determines that it is unsafe when the position of the object does not correspond to a predetermined position of the person performing the predetermined action. The management device according to any one of claims 1 to 7.

9. The computer Detects a specific person's movement from an image taken at a specific location, identifying relevant images in the images that show predetermined objects related to the safety of the person; If the motion in the attention area associated with the related image is a default motion, it is determined that the person performing the motion is in a safe situation; If the motion in the attention area is not a predetermined motion, it is determined that the person performing the motion is not in a safe situation; outputting determination information including the result of the determination; Management method.

10. Detecting a predetermined movement of a person from an image taken at a predetermined location, identifying relevant images in the images that show predetermined objects related to the safety of the person; If the motion in the attention area associated with the related image is a default motion, it is determined that the person performing the motion is in a safe situation; If the motion in the attention area is not a predetermined motion, it is determined that the person performing the motion is not in a safe situation; outputting determination information including the result of the determination; A program that causes a computer to execute a management method.

Images