Person action recognition system and method based on deep neural network

Abstract

The application discloses a personnel action recognition system and method based on a deep neural network, relates to the technical field of industrial safety monitoring, and utilizes a deep neural network to recognize identity information, a line-of-sight direction and a body orientation of a target personnel in a work area; determines an auxiliary monitoring capability of a first device and the target personnel on a second device; obtains a monitoring situation of the target personnel on the first device and an auxiliary monitoring situation of the target personnel on the second device; obtains a monitoring situation of a device in a monitoring area, performs risk assessment on the device in the work area based on the monitoring situation of the device, and performs risk assessment on the device according to the monitoring situation of the personnel on the device when the device is working; the line-of-sight direction of the personnel is recognized by utilizing the deep neural network technology, a work time line of the device and a monitoring time line of the personnel on the device are generated, the risk situation of the device is assessed based on the work time line of the device, and reference can be provided for device maintenance.

Description

Technical Field

[0001] This invention relates to the field of industrial safety monitoring technology, specifically to a personnel action recognition system and method based on deep neural networks. Background Technology

[0002] Industrial equipment risk assessment is a comprehensive, systematic, and scientific method for evaluating the safety of industrial equipment. By collecting, monitoring, and reviewing information on industrial equipment, its safety status and risk level can be determined, enabling the implementation of corresponding safety measures to prevent accidents and ensure the safety of personnel and equipment. Personnel play a crucial role in industrial equipment risk assessment; their experience, knowledge, and operational performance significantly impact the assessment. Therefore, how to monitor and analyze personnel actions in real time, and then assess equipment safety risks based on these actions within the work area, has become a pressing issue. Summary of the Invention

[0003] The purpose of this invention is to provide a human motion recognition system and method based on deep neural networks to solve the problems raised in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a personnel action recognition system based on a deep neural network, comprising an image acquisition module, a data storage module, and an analysis module; the output end of the image acquisition module is connected to the input ends of the data storage module and the analysis module, for acquiring monitoring image data within the work area; the data storage module and the analysis module are interconnected, for storing monitoring image data within the work area, monitoring task data assigned to the personnel, and risk assessment data of the equipment; the analysis module, based on the personnel's monitoring of the equipment and auxiliary monitoring data, obtains the monitoring status of the equipment within the monitoring area, and performs a risk assessment of the equipment within the work area based on the monitoring status of the equipment.

[0005] Specifically, the image acquisition module further includes a neural network unit, a sensor unit, and a communication unit; the neural network unit is used to identify the monitoring images and determine the body orientation and line of sight of personnel; the sensor unit is used to acquire monitoring images of the work area; and the communication unit is used to realize data transmission between the image acquisition module, the data storage module, and the analysis module.

[0006] Specifically, the neural network unit also includes a gaze direction prediction model during the presence of an obstruction. It takes the gaze direction and body orientation features of the target person before and after the presence of the obstruction as input, obtains the gaze direction of the target person during the presence of the obstruction, and obtains the monitoring status of the target person on the device during the presence of the obstruction based on the gaze direction of the target person.

[0007] Specifically, the analysis module further includes a risk value assessment unit, a correlation analysis unit, and a timeline generation unit; the risk value assessment unit assesses the risk of the equipment based on the equipment's operating timeline; the correlation analysis unit analyzes the correlation between equipment and the personnel's auxiliary monitoring capabilities based on historical data of the equipment monitored by personnel; and the timeline generation unit generates the personnel's monitoring timeline and the equipment's operating timeline.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a person action recognition method based on deep neural networks, comprising the following steps:

[0009] Acquire surveillance image data within the work area and use deep neural networks to identify the identity information, line of sight, and body orientation of target personnel within the work area;

[0010] Based on the target personnel's identity information, determine the first device and the target personnel's auxiliary monitoring capabilities for the second device; the first device is the device that the target personnel are responsible for monitoring, and the second device is other equipment in the work area; based on the target personnel's line of sight and body orientation, obtain the target personnel's monitoring status of the first device and their auxiliary monitoring status of the second device; integrate all the target personnel's monitoring status of the first device and their auxiliary monitoring status of the second device to obtain the monitoring status of the equipment in the monitoring area, and conduct a risk assessment of the equipment in the work area based on the equipment monitoring status.

[0011] Specifically, the step of using a deep neural network to identify the identity information, line of sight, and body orientation of target personnel within the work area also includes the following steps:

[0012] The system acquires historical surveillance images of personnel within the work area; extracts personnel identification features, body features, and facial features from these images; uses personnel identification features as input and personnel identity as output to train a deep neural network model for personnel identification; uses personnel body features as input and personnel body orientation as output to train a deep neural network model for personnel body orientation; uses personnel facial features as input and personnel gaze direction as output to train a deep neural network model for personnel gaze direction.

[0013] The trained deep neural network model is used to continuously identify the line of sight and body orientation of the target personnel in the work area. When there is an obstruction between the target personnel and the image acquisition unit, the line of sight and body orientation of the target personnel before the obstruction is present, and the line of sight and body orientation of the target personnel after the obstruction is present are obtained. The body orientation and line of sight of the target personnel during the period when the obstruction is present are analyzed.

[0014] Specifically, determining the auxiliary monitoring capabilities of the first device and the target personnel for the second device based on the target personnel's identity information further includes the following steps:

[0015] Obtain the model information of the first device and the model information of the second device. If the model information of the second device is the same as that of the first device, then the target personnel have full auxiliary monitoring capabilities for the second device. If the model information of the first device and the model information of the second device are different, analyze the correlation between the first device and the second device, including the following steps:

[0016] Obtain information on all personnel within the work area who are responsible for monitoring the equipment, and determine whether there are any reference personnel who have been responsible for monitoring both the first and second equipment. If no reference personnel exist, the target personnel only have basic auxiliary monitoring capabilities for the second equipment.

[0017] If reference personnel exist, their historical monitoring information for the first and second devices is obtained. The monitoring time for the first and second devices is expanded on a timeline, with the start time of monitoring as the starting point and the end time (or the current time) as the end point. The length of the timeline represents the total monitoring time for the first and second devices. The time value T1 for the first device and T2 for the second device are calculated: T1 = ∑(ti - t0), where t0 is the starting point of the timeline and ti is the time point when the reference personnel began monitoring the first device; T2 = ∑(tj - t0), where tj is the time point when the reference personnel began monitoring the second device. The average values ​​AT1 and AT2 of T1 and T2 are obtained from the data of all reference personnel.

[0018] Compare the relationship between AT1 and AT2. If AT1 is greater than AT2, it indicates that the monitoring requirements of the second device are lower than those of the first device, and the target personnel have full auxiliary monitoring capabilities for the second device. If AT1 is less than or equal to AT2, it indicates that the monitoring requirements of the second device are higher than those of the first device. Obtain the time t1 for the reference personnel to monitor the first device and the time t2 for the reference personnel to monitor the second device. Calculate the average interval G between the time the reference personnel are responsible for monitoring the first device and the starting point of the time axis, where G = T1 / t1. Take the data of all reference personnel to obtain the average value AG of G, where AG is the monitoring time of the reference personnel on the first device before they start monitoring the second device. Obtain the monitoring time t of the target personnel on the first device, and obtain the partial auxiliary monitoring capability sk of the target personnel on the second device, where sk = min{1, t / 2 × AG}.

[0019] Specifically, the analysis of a person's body orientation and line of sight during the presence of an obstruction also includes the following steps:

[0020] Generate the initial monitoring timeline of the target personnel for the k-th device. The initial label on the initial monitoring timeline is "no monitoring". Add the monitoring situation of the target personnel for the k-th device when there are no obstructions to the initial monitoring timeline of the target personnel for the k-th device. Change the label of the time corresponding to the target personnel monitoring the k-th device on the initial monitoring timeline to "full monitoring", "partial monitoring" or "basic monitoring", depending on the target personnel's full auxiliary monitoring capability, partial auxiliary monitoring capability or basic auxiliary monitoring capability for the k-th device.

[0021] From the historical data of the work area, extract the gaze direction and body orientation features of historical personnel before and after the existence of the obstruction. Use the extracted gaze direction and body orientation features as input and the actual observation results as output to train the gaze direction prediction model during the existence of the obstruction.

[0022] Using a trained model for predicting gaze direction during the presence of an obstruction, the model inputs the gaze direction and body orientation features of the target person before and after the obstruction. This yields the gaze direction of the target person during the presence of the obstruction. Based on the gaze direction, the model obtains the target person's monitoring status of the k-th device during the presence of the obstruction. The monitoring status of the k-th device during the presence of the obstruction is then added to the target person's monitoring timeline for the k-th device.

[0023] Generate the initial operation timeline of the k-th device, with the initial label being "unmanned monitoring"; obtain the monitoring timeline of the h-th person for the k-th device within the operation area, and overlay the monitoring timeline of the h-th person for the k-th device with the initial operation timeline of the k-th device, covering the labels according to their priority, with the label priority in descending order as: full monitoring, partial monitoring, basic monitoring, and unmanned monitoring; overlay the monitoring timelines of all personnel for the k-th device within the operation area to obtain the operation timeline of the k-th device.

[0024] Obtain the operation timeline of the k-th device within the work area, and calculate the risk value Fk of the k-th device based on the timeline, where Fk = W1×Y1 + W2×Y2 + W3×∑ h (∫sk h dt)+W4×Y4, where Y1, Y3, and Y4 represent the time for full monitoring, basic monitoring, and unattended monitoring during the operation of the k-th device, and W1, W2, W3, and W4 are the weights, ∫sk h dt represents the auxiliary monitoring time of the h-th person on the k-th device within the work area, and sk h This represents the partial auxiliary monitoring capability of the h-th person in the work area for the k-th equipment.

[0025] Compared with the prior art, the beneficial effects of the present invention are: the risk of the equipment is assessed based on the monitoring of the equipment by personnel during equipment operation; the direction of personnel's line of sight is identified by deep neural network technology, generating the equipment operation timeline and the personnel's monitoring timeline of the equipment, and the risk of the equipment is assessed based on the equipment operation timeline, which can provide a reference for equipment maintenance. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the structure of the deep neural network-based human motion recognition system of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Example: Figure 1 As shown, this invention provides a technical solution: a personnel action recognition system based on a deep neural network, comprising an image acquisition module, a data storage module, and an analysis module. The output of the image acquisition module is connected to the input of the data storage module and the analysis module, used to acquire monitoring image data within the work area. The data storage module and the analysis module are interconnected, used to store monitoring image data within the work area, monitoring task data assigned to personnel, and risk assessment data of equipment. The analysis module, based on personnel monitoring and auxiliary monitoring of equipment, obtains the monitoring status of equipment within the monitoring area, and performs risk assessment of equipment within the work area based on the monitoring status.

[0029] Specifically, the image acquisition module further includes a neural network unit, a sensor unit, and a communication unit; the neural network unit is used to identify the monitoring images and determine the body orientation and line of sight of personnel; the sensor unit is used to acquire monitoring images of the work area; and the communication unit is used to realize data transmission between the image acquisition module, the data storage module, and the analysis module.

[0030] Specifically, the neural network unit also includes a gaze direction prediction model during the presence of an obstruction. It takes the gaze direction and body orientation features of the target person before and after the presence of the obstruction as input, obtains the gaze direction of the target person during the presence of the obstruction, and obtains the monitoring status of the target person on the device during the presence of the obstruction based on the gaze direction of the target person.

[0031] Specifically, the analysis module further includes a risk value assessment unit, a correlation analysis unit, and a timeline generation unit; the risk value assessment unit assesses the risk of the equipment based on the equipment's operating timeline; the correlation analysis unit analyzes the correlation between equipment and the personnel's auxiliary monitoring capabilities based on historical data of the equipment monitored by personnel; and the timeline generation unit generates the personnel's monitoring timeline and the equipment's operating timeline.

[0032] Example: This invention provides a technical solution, a method for human action recognition based on deep neural networks, comprising the following steps:

[0033] Acquire surveillance image data within the work area and use deep neural networks to identify the identity information, line of sight, and body orientation of target personnel within the work area;

[0034] The system acquires historical surveillance images of personnel within the work area; extracts personnel identification features, body features, and facial features from these images; uses personnel identification features as input and personnel identity as output to train a deep neural network model for personnel identification; uses personnel body features as input and personnel body orientation as output to train a deep neural network model for personnel body orientation; uses personnel facial features as input and personnel gaze direction as output to train a deep neural network model for personnel gaze direction.

[0035] The identification characteristics of individuals include, but are not limited to, clothing style and work identification documents. Using surveillance images containing these characteristics as input and the individual's identity as output, the identification can be quantified by assigning labels to the target individuals. A convolutional neural network model for identity information is trained. A network structure consisting of an input layer, convolutional layer, pooling layer, activation function layer, fully connected layer, and output layer is established. The surveillance image containing the identification characteristics is input into the input layer of the convolutional neural network, and a regression value is obtained from the output layer. A threshold is used to determine the label corresponding to the regression value, thus identifying the individual's identity information. Surveillance images with known body orientation are acquired. The individual's body characteristics include, but are not limited to, the orientation of the head, torso, shoes, and clothing. Using surveillance images containing this information as input and the known body orientation as output, a deep neural network model for body orientation can be trained. For body orientation, a certain direction can be set as the zero point, allowing the deviation of the individual from the zero point direction to obtain the individual's body orientation information, thus quantifying the body orientation. Facial features of individuals include, but are not limited to, the eyes and head. The specific training method for the deep neural network model for gaze direction is the same as that for the deep neural network model for body orientation.

[0036] The trained deep neural network model is used to continuously identify the line of sight and body orientation of the target personnel in the work area. When there is an obstruction between the target personnel and the image acquisition unit, the line of sight and body orientation of the target personnel before the obstruction is present, and the line of sight and body orientation of the target personnel after the obstruction is present are obtained. The body orientation and line of sight of the target personnel during the period when the obstruction is present are analyzed.

[0037] The image of the target person may be obscured by others or objects due to their passing by or carrying goods, which may affect the recognition of the target person's actions.

[0038] Generate the initial monitoring timeline of the target personnel for the k-th device. The initial label on the initial monitoring timeline is "no monitoring". Add the monitoring situation of the target personnel for the k-th device when there are no obstructions to the initial monitoring timeline of the target personnel for the k-th device. Change the label of the time corresponding to the target personnel monitoring the k-th device on the initial monitoring timeline to "full monitoring", "partial monitoring" or "basic monitoring", depending on the target personnel's full auxiliary monitoring capability, partial auxiliary monitoring capability or basic auxiliary monitoring capability for the k-th device.

[0039] From the historical data of the work area, extract the gaze direction and body orientation features of historical personnel before and after the existence of the obstruction. Use the extracted gaze direction and body orientation features as input and the actual observation results as output to train the gaze direction prediction model during the existence of the obstruction.

[0040] Using a trained model for predicting gaze direction during the presence of an obstruction, the model inputs the gaze direction and body orientation features of the target person before and after the obstruction. This yields the gaze direction of the target person during the presence of the obstruction. Based on the gaze direction, the model obtains the target person's monitoring status of the k-th device during the presence of the obstruction. The monitoring status of the k-th device during the presence of the obstruction is then added to the target person's monitoring timeline for the k-th device.

[0041] This study utilizes the body orientation and gaze direction features of historical individuals in several frames before and after the appearance of an occlusion using a convolutional neural network to obtain time series of these features. The image frame data is divided into training and testing sets. Using the time series of body orientation and gaze direction features from the training set as input, and the actual gaze direction of individuals during the occlusion period as output, a gaze direction prediction model for the occlusion period is trained. The parameters of the prediction model are updated using an error backpropagation algorithm. An appropriate loss function is selected through testing to measure the model's prediction error, including but not limited to cross-entropy loss and mean squared error loss. The prediction model is validated using a test set. During this process, the actual gaze direction of individuals during the occlusion period can be determined by other individuals or by considering their gaze direction and body orientation before and after the occlusion. For example, if an individual's body orientation and gaze direction remain consistent before and after the occlusion, and their body orientation is the same before and after the occlusion, then it is highly likely that their gaze direction will not change during the occlusion period. The actual line of sight of personnel during the period when the obstruction exists; or, if the obstruction exists for a short time and the personnel's line of sight did not change or changed only slightly before the obstruction appeared, then the line of sight is very likely not to change; if the personnel's body orientation and line of sight were inconsistent before the obstruction appeared, but became consistent after the obstruction appeared, then the personnel's line of sight is very likely to move towards the body orientation; training is conducted using image frame data of the actual line of sight of personnel during the period when the obstruction exists; in most cases, the personnel's line of sight and body orientation do indeed remain consistent. For example, when personnel are focused on observing a target, in the work area, i.e., when monitoring the equipment under their responsibility, they will usually naturally face the target and look at the target to obtain a better visual effect and communication experience. At this time, the body orientation and line of sight are the same; when walking or moving normally, personnel will face the direction of travel, and their eyes will also be focused on the road ahead. At this time, the body orientation and line of sight are the same; personnel can provide some auxiliary monitoring capabilities for passing equipment, depending on the personnel's understanding of the equipment and whether they have observed the equipment.

[0042] Based on the target personnel's identity information, determine the first device and the target personnel's auxiliary monitoring capabilities on the second device; the first device is the device that the target personnel are responsible for monitoring, and the second device is other equipment within the work area; based on the target personnel's line of sight and body orientation, obtain the target personnel's monitoring status of the first device and their auxiliary monitoring status of the second device;

[0043] Obtain the model information of the first device and the model information of the second device. If the model information of the second device is the same as that of the first device, then the target personnel have full auxiliary monitoring capabilities for the second device. If the model information of the first device and the model information of the second device are different, analyze the correlation between the first device and the second device, including the following steps:

[0044] Obtain information on all personnel within the work area who are responsible for monitoring the equipment, and determine whether there are any reference personnel who have been responsible for monitoring both the first and second equipment. If no reference personnel exist, the target personnel only have basic auxiliary monitoring capabilities for the second equipment.

[0045] If reference personnel exist, their historical monitoring information for the first and second devices is obtained. The monitoring time for the first and second devices is expanded on a timeline, with the start time of monitoring as the starting point and the end time (or the current time) as the end point. The length of the timeline represents the total monitoring time for the first and second devices. The time value T1 for the first device and T2 for the second device are calculated: T1 = ∑(ti - t0), where t0 is the starting point of the timeline and ti is the time point when the reference personnel began monitoring the first device; T2 = ∑(tj - t0), where tj is the time point when the reference personnel began monitoring the second device. The average values ​​AT1 and AT2 of T1 and T2 are obtained from the data of all reference personnel.

[0046] Compare the relationship between AT1 and AT2. If AT1 is greater than AT2, it indicates that the monitoring requirements of the second device are lower than those of the first device, and the target personnel have full auxiliary monitoring capabilities for the second device. If AT1 is less than or equal to AT2, it indicates that the monitoring requirements of the second device are higher than those of the first device. Obtain the time t1 for the reference personnel to monitor the first device and the time t2 for the reference personnel to monitor the second device. Calculate the average interval G between the time the reference personnel are responsible for monitoring the first device and the starting point of the time axis, where G = T1 / t1. Take the data of all reference personnel to obtain the average value AG of G, where AG is the monitoring time of the reference personnel on the first device before they start monitoring the second device. Obtain the monitoring time t of the target personnel on the first device, and obtain the partial auxiliary monitoring capability sk of the target personnel on the second device, where sk = min{1, t / 2 × AG}.

[0047] First, determine if there are any personnel who have been in charge of both the first and second devices. If not, it means that the first and second devices are unrelated, and the target personnel lack the necessary understanding of the second device. They will find it difficult to detect minor problems or anomalies in the second device and can only serve as a warning when there are obvious problems with the second device.

[0048] When reference personnel are available, it is necessary to determine whether there is a progressive relationship between the first and second devices, i.e., whether the monitoring of the second device depends on the monitoring experience of the first device. Therefore, the time data of the reference personnel responsible for monitoring the second and first devices should be obtained to obtain the time distribution of the reference personnel's monitoring of the second and first devices. For example, if the reference personnel were responsible for monitoring the first device in the previous month and the second device in the following month, and the unit of time is days, integrating the time interval from 0 to 30 will give the time value T1 = 450 for the first device, and integrating the time interval from 30 to 60 will give the time value T2 = 1350 for the second device. The distribution of time values ​​T1 and T2 can be obtained from the data of all reference personnel. If T2 is greater than T1, there is a progressive relationship between the first and second devices, and vice versa. Based on the current monitoring time of the target personnel on the first device and the time required to monitor the second device, the target personnel's auxiliary monitoring capability on the second device can be determined. For example, when T1 and T2 are 450 and 1350 respectively, the average interval G between the time of the first device and the starting point of the time axis is calculated to be 15, and 2×G=30, where 30 is the time when the reference personnel begin to monitor the second device. The time when the reference personnel are responsible for monitoring the first and second devices can be distributed alternately. In this case, the time interval can be integrated segment by segment. Depending on the specific requirements of the equipment, the unit of time can also be hours, minutes, etc.

[0049] By integrating the monitoring data of all target personnel on the first equipment and the auxiliary monitoring data on the second equipment, the monitoring data of the equipment within the monitoring area is obtained. Based on the monitoring data, a risk assessment is conducted on the equipment within the work area.

[0050] Generate the initial operation timeline of the k-th device, with the initial label being "unmanned monitoring"; obtain the monitoring timeline of the h-th person for the k-th device within the operation area, and overlay the monitoring timeline of the h-th person for the k-th device with the initial operation timeline of the k-th device, covering the labels according to their priority, with the label priority in descending order as: full monitoring, partial monitoring, basic monitoring, and unmanned monitoring; overlay the monitoring timelines of all personnel for the k-th device within the operation area to obtain the operation timeline of the k-th device.

[0051] Obtain the operation timeline of the k-th device within the work area, and calculate the risk value Fk of the k-th device based on the timeline, where Fk = W1×Y1 + W2×Y2 + W3×∑ h (∫sk h dt)+W4×Y4, where Y1, Y3, and Y4 represent the time for full monitoring, basic monitoring, and unattended monitoring during the operation of the k-th device, and W1, W2, W3, and W4 are the weights, ∫sk h dt represents the auxiliary monitoring time of the h-th person on the k-th device within the work area, and sk h This represents the partial auxiliary monitoring capability of the h-th person in the work area for the k-th equipment.

[0052] Equipment failure risk is related to the equipment's operating time. When someone monitors the equipment, if the person has full monitoring capability (personnel are responsible for monitoring the equipment) or full auxiliary monitoring capability, they can detect minor anomalies or abnormalities that randomly occur during equipment operation, or adjust equipment parameters in real time, thereby mitigating the risk of equipment failure. If the person has partial auxiliary monitoring capability, they may or may not notice minor anomalies or abnormalities, depending on their understanding of the equipment. The partial auxiliary monitoring capability sk of personnel in the work area changes with time t. When personnel only have basic auxiliary monitoring capability, due to insufficient understanding of the equipment, it can only serve as a warning when there are very obvious anomalies. Specifically, tags with the same priority are retained on the equipment's operating line simultaneously. For example, if the partial auxiliary monitoring time of personnel h1 and h2 on equipment k overlaps during the day the k-th equipment is operating, then the partial auxiliary monitoring capability of the overlapping part is 1 - (1 - sk). h1 (1-sk) h2 ), sk h1 and SK h2 This represents the partial auxiliary monitoring capability of the h1th and h2th personnel for the kth device. In the overlapping portion, 1-(1-sk) h1 (1-sk) h2 ) Replace sk h Calculations are performed, and sk is used for the non-overlapping parts. h Perform the calculation.

[0053] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for recognizing human actions based on deep neural networks, characterized in that, Includes the following steps: Acquire surveillance image data within the work area and use deep neural networks to identify the identity information, line of sight, and body orientation of target personnel within the work area; Based on the target personnel's identity information, determine the first device and the target personnel's auxiliary monitoring capabilities for the second device; the first device is the device monitored by the target personnel, and the second device is other equipment within the work area; based on the target personnel's line of sight and body orientation, obtain the target personnel's monitoring status of the first device and their auxiliary monitoring status of the second device; integrate all the target personnel's monitoring status of the first device and their auxiliary monitoring status of the second device to obtain the monitoring status of the equipment within the monitoring area, and conduct a risk assessment of the equipment within the work area based on the equipment monitoring status; The process of determining the auxiliary monitoring capabilities of the first device and the target personnel for the second device based on the target personnel's identity information also includes the following steps: Obtain the model information of the first device and the model information of the second device. If the model information of the second device is the same as that of the first device, then the target personnel have full auxiliary monitoring capabilities for the second device. If the model information of the first device and the model information of the second device are different, analyze the correlation between the first device and the second device, including the following steps: Obtain information on all personnel within the work area who are responsible for monitoring the equipment, and determine whether there are any reference personnel who have been responsible for monitoring both the first and second equipment. If no reference personnel exist, the target personnel only have basic auxiliary monitoring capabilities for the second equipment. If reference personnel exist, their historical monitoring information for the first and second devices is obtained. The monitoring time for the first and second devices is expanded on a timeline, with the start time of monitoring as the starting point and the end time (or the current time) as the end point. The length of the timeline represents the total monitoring time for the first and second devices. The time value T1 for the first device and T2 for the second device are calculated: T1 = ∑(ti - t0), where t0 is the starting point of the timeline and ti is the time point when the reference personnel began monitoring the first device; T2 = ∑(tj - t0), where tj is the time point when the reference personnel began monitoring the second device. The average values ​​AT1 and AT2 of T1 and T2 are obtained from the data of all reference personnel. Compare the relationship between AT1 and AT2. If AT1 is greater than AT2, it indicates that the monitoring requirements of the second device are lower than those of the first device, and the target personnel have full auxiliary monitoring capabilities for the second device. If AT1 is less than or equal to AT2, it indicates that the monitoring requirements of the second device are higher than those of the first device. Obtain the time t1 for the reference personnel to monitor the first device and the time t2 for the reference personnel to monitor the second device. Calculate the average interval G between the time the reference personnel are responsible for monitoring the first device and the starting point of the time axis, where G = T1 / t1. Take the data of all reference personnel to obtain the average value AG of G, where AG is the monitoring time of the reference personnel on the first device before they start monitoring the second device. Obtain the monitoring time t of the target personnel on the first device, and obtain the partial auxiliary monitoring capability sk of the target personnel on the second device, where sk = min{1, t / 2 × AG}.

2. The method for human action recognition based on deep neural networks according to claim 1, characterized in that, The process of using deep neural networks to identify the identity information, line of sight, and body orientation of target personnel within the work area also includes the following steps: The system acquires historical surveillance images of personnel within the work area; extracts personnel identification features, body features, and facial features from these images; uses personnel identification features as input and personnel identity as output to train a deep neural network model for personnel identification; uses personnel body features as input and personnel body orientation as output to train a deep neural network model for personnel body orientation; uses personnel facial features as input and personnel gaze direction as output to train a deep neural network model for personnel gaze direction. The trained deep neural network model is used to continuously identify the line of sight and body orientation of the target personnel in the work area. When there is an obstruction between the target personnel and the image acquisition unit, the line of sight and body orientation of the target personnel before the obstruction is present and the line of sight and body orientation of the target personnel after the obstruction is present are obtained. The body orientation and line of sight of the target personnel during the period when the obstruction is present are analyzed.

3. The method for human action recognition based on deep neural networks according to claim 2, characterized in that, The analysis of a person's body orientation and line of sight during the presence of an obstruction also includes the following steps: Generate the initial monitoring timeline of the target personnel for the k-th device. The initial label on the initial monitoring timeline is "no monitoring". Add the monitoring situation of the target personnel for the k-th device when there are no obstructions to the initial monitoring timeline of the target personnel for the k-th device. Change the label of the time corresponding to the target personnel monitoring the k-th device on the initial monitoring timeline to "full monitoring", "partial monitoring" or "basic monitoring", depending on the target personnel's full auxiliary monitoring capability, partial auxiliary monitoring capability or basic auxiliary monitoring capability for the k-th device. From the historical data of the work area, extract the gaze direction and body orientation features of historical personnel before and after the existence of the obstruction. Use the extracted gaze direction and body orientation features as input and the actual observation results as output to train the gaze direction prediction model during the existence of the obstruction. Using a trained model for predicting gaze direction during the presence of an obstruction, the model inputs the gaze direction and body orientation features of the target person before and after the obstruction. This yields the gaze direction of the target person during the presence of the obstruction. Based on the gaze direction, the model obtains the target person's monitoring status of the k-th device during the presence of the obstruction. The monitoring status of the k-th device during the presence of the obstruction is then added to the target person's monitoring timeline for the k-th device.

4. The human action recognition method based on deep neural networks according to claim 3, characterized in that, The risk assessment of equipment within the work area based on equipment monitoring also includes the following steps. Specifically, the following steps are included: Generate the initial operation timeline of the k-th device, with the initial label being "unmanned monitoring"; obtain the monitoring timeline of the h-th person for the k-th device within the operation area, and overlay the monitoring timeline of the h-th person for the k-th device with the initial operation timeline of the k-th device, covering the labels according to their priority, with the label priority in descending order as: full monitoring, partial monitoring, basic monitoring, and unmanned monitoring; overlay the monitoring timelines of all personnel for the k-th device within the operation area to obtain the operation timeline of the k-th device.

5. The human action recognition method based on deep neural networks according to claim 4, characterized in that, The risk assessment of equipment within the work area based on equipment monitoring also includes the following steps: Obtain the operation timeline of the k-th device within the work area, and calculate the risk value Fk of the k-th device based on the timeline. In the formula, Y1, Y3, and Y4 represent the time for full monitoring, basic monitoring, and unattended monitoring when the k-th device is operating, and W1, W2, W3, and W4 are the weights. sk represents the auxiliary monitoring time of the h-th person on the k-th device within the work area. h This represents the partial auxiliary monitoring capability of the h-th person in the work area for the k-th equipment.

6. A human motion recognition system based on deep neural networks, characterized in that, It includes an image acquisition module, a data storage module, and an analysis module; the output of the image acquisition module is connected to the input of the data storage module and the analysis module, and is used to acquire monitoring image data within the work area; the data storage module is interconnected with the analysis module and is used to store monitoring image data within the work area, monitoring task data of the operators, and risk assessment data of the equipment. The analysis module obtains the monitoring status of the equipment within the monitoring area based on personnel monitoring and auxiliary monitoring, and conducts risk assessment of the equipment within the work area based on the monitoring status.

7. The personnel action recognition system based on a deep neural network according to claim 6, characterized in that, The image acquisition module further includes a neural network unit, a sensor unit, and a communication unit; the neural network unit is used to identify the monitored images and determine the body orientation and line of sight of personnel; the sensor unit is used to acquire monitored images of the work area; and the communication unit is used to realize data transmission between the image acquisition module, the data storage module, and the analysis module.

8. The personnel action recognition system based on a deep neural network according to claim 7, characterized in that, The neural network unit also includes a line-of-sight prediction model during the presence of an obstruction. It takes the line-of-sight and body orientation features of the target person before and after the obstruction is present as input, obtains the line-of-sight of the target person during the presence of the obstruction, and obtains the monitoring status of the target person on the device during the presence of the obstruction based on the line-of-sight of the target person.

9. The personnel action recognition system based on a deep neural network according to claim 8, characterized in that, The analysis module further includes a risk value assessment unit, a correlation analysis unit, and a timeline generation unit; the risk value assessment unit assesses the risk of the equipment based on its operating timeline; the correlation analysis unit analyzes the correlation between equipment and the personnel's auxiliary monitoring capabilities based on historical data of the equipment monitored by personnel; the timeline generation unit generates the monitoring timeline of the equipment by personnel and the operating timeline of the equipment.

Images